<!DOCTYPE html><html><head>
      <title>supp-matrix-calculus</title>
      <meta charset="utf-8">
      <meta name="viewport" content="width=device-width, initial-scale=1.0">
      
      <style>
      /*!
* reveal.js 4.0.2
* https://revealjs.com
* MIT licensed
*
* Copyright (C) 2020 Hakim El Hattab, https://hakim.se
*/
.reveal .r-stretch,.reveal .stretch{max-width:none;max-height:none}.reveal pre.r-stretch code,.reveal pre.stretch code{height:100%;max-height:100%;box-sizing:border-box}.reveal .r-fit-text{display:inline-block;white-space:nowrap}.reveal .r-stack{display:grid}.reveal .r-stack>*{grid-area:1/1;margin:auto}.reveal .r-hstack,.reveal .r-vstack{display:flex}.reveal .r-hstack img,.reveal .r-hstack video,.reveal .r-vstack img,.reveal .r-vstack video{min-width:0;min-height:0;-o-object-fit:contain;object-fit:contain}.reveal .r-vstack{flex-direction:column;align-items:center;justify-content:center}.reveal .r-hstack{flex-direction:row;align-items:center;justify-content:center}.reveal .items-stretch{align-items:stretch}.reveal .items-start{align-items:flex-start}.reveal .items-center{align-items:center}.reveal .items-end{align-items:flex-end}.reveal .justify-between{justify-content:space-between}.reveal .justify-around{justify-content:space-around}.reveal .justify-start{justify-content:flex-start}.reveal .justify-center{justify-content:center}.reveal .justify-end{justify-content:flex-end}html.reveal-full-page{width:100%;height:100%;height:100vh;height:calc(var(--vh,1vh) * 100);overflow:hidden}.reveal-viewport{height:100%;overflow:hidden;position:relative;line-height:1;margin:0;background-color:#fff;color:#000}.reveal .slides section .fragment{opacity:0;visibility:hidden;transition:all .2s ease;will-change:opacity}.reveal .slides section .fragment.visible{opacity:1;visibility:inherit}.reveal .slides section .fragment.disabled{transition:none}.reveal .slides section .fragment.grow{opacity:1;visibility:inherit}.reveal .slides section .fragment.grow.visible{transform:scale(1.3)}.reveal .slides section .fragment.shrink{opacity:1;visibility:inherit}.reveal .slides section .fragment.shrink.visible{transform:scale(.7)}.reveal .slides section .fragment.zoom-in{transform:scale(.1)}.reveal .slides section .fragment.zoom-in.visible{transform:none}.reveal .slides section .fragment.fade-out{opacity:1;visibility:inherit}.reveal .slides section .fragment.fade-out.visible{opacity:0;visibility:hidden}.reveal .slides section .fragment.semi-fade-out{opacity:1;visibility:inherit}.reveal .slides section .fragment.semi-fade-out.visible{opacity:.5;visibility:inherit}.reveal .slides section .fragment.strike{opacity:1;visibility:inherit}.reveal .slides section .fragment.strike.visible{text-decoration:line-through}.reveal .slides section .fragment.fade-up{transform:translate(0,40px)}.reveal .slides section .fragment.fade-up.visible{transform:translate(0,0)}.reveal .slides section .fragment.fade-down{transform:translate(0,-40px)}.reveal .slides section .fragment.fade-down.visible{transform:translate(0,0)}.reveal .slides section .fragment.fade-right{transform:translate(-40px,0)}.reveal .slides section .fragment.fade-right.visible{transform:translate(0,0)}.reveal .slides section .fragment.fade-left{transform:translate(40px,0)}.reveal .slides section .fragment.fade-left.visible{transform:translate(0,0)}.reveal .slides section .fragment.current-visible,.reveal .slides section .fragment.fade-in-then-out{opacity:0;visibility:hidden}.reveal .slides section .fragment.current-visible.current-fragment,.reveal .slides section .fragment.fade-in-then-out.current-fragment{opacity:1;visibility:inherit}.reveal .slides section .fragment.fade-in-then-semi-out{opacity:0;visibility:hidden}.reveal .slides section .fragment.fade-in-then-semi-out.visible{opacity:.5;visibility:inherit}.reveal .slides section .fragment.fade-in-then-semi-out.current-fragment{opacity:1;visibility:inherit}.reveal .slides section .fragment.highlight-blue,.reveal .slides section .fragment.highlight-current-blue,.reveal .slides section .fragment.highlight-current-green,.reveal .slides section .fragment.highlight-current-red,.reveal .slides section .fragment.highlight-green,.reveal .slides section .fragment.highlight-red{opacity:1;visibility:inherit}.reveal .slides section .fragment.highlight-red.visible{color:#ff2c2d}.reveal .slides section .fragment.highlight-green.visible{color:#17ff2e}.reveal .slides section .fragment.highlight-blue.visible{color:#1b91ff}.reveal .slides section .fragment.highlight-current-red.current-fragment{color:#ff2c2d}.reveal .slides section .fragment.highlight-current-green.current-fragment{color:#17ff2e}.reveal .slides section .fragment.highlight-current-blue.current-fragment{color:#1b91ff}.reveal:after{content:'';font-style:italic}.reveal iframe{z-index:1}.reveal a{position:relative}@keyframes bounce-right{0%,10%,25%,40%,50%{transform:translateX(0)}20%{transform:translateX(10px)}30%{transform:translateX(-5px)}}@keyframes bounce-left{0%,10%,25%,40%,50%{transform:translateX(0)}20%{transform:translateX(-10px)}30%{transform:translateX(5px)}}@keyframes bounce-down{0%,10%,25%,40%,50%{transform:translateY(0)}20%{transform:translateY(10px)}30%{transform:translateY(-5px)}}.reveal .controls{display:none;position:absolute;top:auto;bottom:12px;right:12px;left:auto;z-index:11;color:#000;pointer-events:none;font-size:10px}.reveal .controls button{position:absolute;padding:0;background-color:transparent;border:0;outline:0;cursor:pointer;color:currentColor;transform:scale(.9999);transition:color .2s ease,opacity .2s ease,transform .2s ease;z-index:2;pointer-events:auto;font-size:inherit;visibility:hidden;opacity:0;-webkit-appearance:none;-webkit-tap-highlight-color:transparent}.reveal .controls .controls-arrow:after,.reveal .controls .controls-arrow:before{content:'';position:absolute;top:0;left:0;width:2.6em;height:.5em;border-radius:.25em;background-color:currentColor;transition:all .15s ease,background-color .8s ease;transform-origin:.2em 50%;will-change:transform}.reveal .controls .controls-arrow{position:relative;width:3.6em;height:3.6em}.reveal .controls .controls-arrow:before{transform:translateX(.5em) translateY(1.55em) rotate(45deg)}.reveal .controls .controls-arrow:after{transform:translateX(.5em) translateY(1.55em) rotate(-45deg)}.reveal .controls .controls-arrow:hover:before{transform:translateX(.5em) translateY(1.55em) rotate(40deg)}.reveal .controls .controls-arrow:hover:after{transform:translateX(.5em) translateY(1.55em) rotate(-40deg)}.reveal .controls .controls-arrow:active:before{transform:translateX(.5em) translateY(1.55em) rotate(36deg)}.reveal .controls .controls-arrow:active:after{transform:translateX(.5em) translateY(1.55em) rotate(-36deg)}.reveal .controls .navigate-left{right:6.4em;bottom:3.2em;transform:translateX(-10px)}.reveal .controls .navigate-left.highlight{animation:bounce-left 2s 50 both ease-out}.reveal .controls .navigate-right{right:0;bottom:3.2em;transform:translateX(10px)}.reveal .controls .navigate-right .controls-arrow{transform:rotate(180deg)}.reveal .controls .navigate-right.highlight{animation:bounce-right 2s 50 both ease-out}.reveal .controls .navigate-up{right:3.2em;bottom:6.4em;transform:translateY(-10px)}.reveal .controls .navigate-up .controls-arrow{transform:rotate(90deg)}.reveal .controls .navigate-down{right:3.2em;bottom:-1.4em;padding-bottom:1.4em;transform:translateY(10px)}.reveal .controls .navigate-down .controls-arrow{transform:rotate(-90deg)}.reveal .controls .navigate-down.highlight{animation:bounce-down 2s 50 both ease-out}.reveal .controls[data-controls-back-arrows=faded] .navigate-up.enabled{opacity:.3}.reveal .controls[data-controls-back-arrows=faded] .navigate-up.enabled:hover{opacity:1}.reveal .controls[data-controls-back-arrows=hidden] .navigate-up.enabled{opacity:0;visibility:hidden}.reveal .controls .enabled{visibility:visible;opacity:.9;cursor:pointer;transform:none}.reveal .controls .enabled.fragmented{opacity:.5}.reveal .controls .enabled.fragmented:hover,.reveal .controls .enabled:hover{opacity:1}.reveal:not(.rtl) .controls[data-controls-back-arrows=faded] .navigate-left.enabled{opacity:.3}.reveal:not(.rtl) .controls[data-controls-back-arrows=faded] .navigate-left.enabled:hover{opacity:1}.reveal:not(.rtl) .controls[data-controls-back-arrows=hidden] .navigate-left.enabled{opacity:0;visibility:hidden}.reveal.rtl .controls[data-controls-back-arrows=faded] .navigate-right.enabled{opacity:.3}.reveal.rtl .controls[data-controls-back-arrows=faded] .navigate-right.enabled:hover{opacity:1}.reveal.rtl .controls[data-controls-back-arrows=hidden] .navigate-right.enabled{opacity:0;visibility:hidden}.reveal[data-navigation-mode=linear].has-horizontal-slides .navigate-down,.reveal[data-navigation-mode=linear].has-horizontal-slides .navigate-up{display:none}.reveal:not(.has-vertical-slides) .controls .navigate-left,.reveal[data-navigation-mode=linear].has-horizontal-slides .navigate-left{bottom:1.4em;right:5.5em}.reveal:not(.has-vertical-slides) .controls .navigate-right,.reveal[data-navigation-mode=linear].has-horizontal-slides .navigate-right{bottom:1.4em;right:.5em}.reveal:not(.has-horizontal-slides) .controls .navigate-up{right:1.4em;bottom:5em}.reveal:not(.has-horizontal-slides) .controls .navigate-down{right:1.4em;bottom:.5em}.reveal.has-dark-background .controls{color:#fff}.reveal.has-light-background .controls{color:#000}.reveal.no-hover .controls .controls-arrow:active:before,.reveal.no-hover .controls .controls-arrow:hover:before{transform:translateX(.5em) translateY(1.55em) rotate(45deg)}.reveal.no-hover .controls .controls-arrow:active:after,.reveal.no-hover .controls .controls-arrow:hover:after{transform:translateX(.5em) translateY(1.55em) rotate(-45deg)}@media screen and (min-width:500px){.reveal .controls[data-controls-layout=edges]{top:0;right:0;bottom:0;left:0}.reveal .controls[data-controls-layout=edges] .navigate-down,.reveal .controls[data-controls-layout=edges] .navigate-left,.reveal .controls[data-controls-layout=edges] .navigate-right,.reveal .controls[data-controls-layout=edges] .navigate-up{bottom:auto;right:auto}.reveal .controls[data-controls-layout=edges] .navigate-left{top:50%;left:.8em;margin-top:-1.8em}.reveal .controls[data-controls-layout=edges] .navigate-right{top:50%;right:.8em;margin-top:-1.8em}.reveal .controls[data-controls-layout=edges] .navigate-up{top:.8em;left:50%;margin-left:-1.8em}.reveal .controls[data-controls-layout=edges] .navigate-down{bottom:-.3em;left:50%;margin-left:-1.8em}}.reveal .progress{position:absolute;display:none;height:3px;width:100%;bottom:0;left:0;z-index:10;background-color:rgba(0,0,0,.2);color:#fff}.reveal .progress:after{content:'';display:block;position:absolute;height:10px;width:100%;top:-10px}.reveal .progress span{display:block;height:100%;width:100%;background-color:currentColor;transition:transform .8s cubic-bezier(.26,.86,.44,.985);transform-origin:0 0;transform:scaleX(0)}.reveal .slide-number{position:absolute;display:block;right:8px;bottom:8px;z-index:31;font-family:Helvetica,sans-serif;font-size:12px;line-height:1;color:#fff;background-color:rgba(0,0,0,.4);padding:5px}.reveal .slide-number a{color:currentColor}.reveal .slide-number-delimiter{margin:0 3px}.reveal{position:relative;width:100%;height:100%;overflow:hidden;touch-action:pinch-zoom}.reveal.embedded{touch-action:pan-y}.reveal .slides{position:absolute;width:100%;height:100%;top:0;right:0;bottom:0;left:0;margin:auto;pointer-events:none;overflow:visible;z-index:1;text-align:center;perspective:600px;perspective-origin:50% 40%}.reveal .slides>section{perspective:600px}.reveal .slides>section,.reveal .slides>section>section{display:none;position:absolute;width:100%;pointer-events:auto;z-index:10;transform-style:flat;transition:transform-origin .8s cubic-bezier(.26,.86,.44,.985),transform .8s cubic-bezier(.26,.86,.44,.985),visibility .8s cubic-bezier(.26,.86,.44,.985),opacity .8s cubic-bezier(.26,.86,.44,.985)}.reveal[data-transition-speed=fast] .slides section{transition-duration:.4s}.reveal[data-transition-speed=slow] .slides section{transition-duration:1.2s}.reveal .slides section[data-transition-speed=fast]{transition-duration:.4s}.reveal .slides section[data-transition-speed=slow]{transition-duration:1.2s}.reveal .slides>section.stack{padding-top:0;padding-bottom:0;pointer-events:none;height:100%}.reveal .slides>section.present,.reveal .slides>section>section.present{display:block;z-index:11;opacity:1}.reveal .slides>section:empty,.reveal .slides>section>section:empty,.reveal .slides>section>section[data-background-interactive],.reveal .slides>section[data-background-interactive]{pointer-events:none}.reveal.center,.reveal.center .slides,.reveal.center .slides section{min-height:0!important}.reveal .slides>section:not(.present),.reveal .slides>section>section:not(.present){pointer-events:none}.reveal.overview .slides>section,.reveal.overview .slides>section>section{pointer-events:auto}.reveal .slides>section.future,.reveal .slides>section.past,.reveal .slides>section>section.future,.reveal .slides>section>section.past{opacity:0}.reveal.slide section{-webkit-backface-visibility:hidden;backface-visibility:hidden}.reveal .slides>section[data-transition=slide].past,.reveal .slides>section[data-transition~=slide-out].past,.reveal.slide .slides>section:not([data-transition]).past{transform:translate(-150%,0)}.reveal .slides>section[data-transition=slide].future,.reveal .slides>section[data-transition~=slide-in].future,.reveal.slide .slides>section:not([data-transition]).future{transform:translate(150%,0)}.reveal .slides>section>section[data-transition=slide].past,.reveal .slides>section>section[data-transition~=slide-out].past,.reveal.slide .slides>section>section:not([data-transition]).past{transform:translate(0,-150%)}.reveal .slides>section>section[data-transition=slide].future,.reveal .slides>section>section[data-transition~=slide-in].future,.reveal.slide .slides>section>section:not([data-transition]).future{transform:translate(0,150%)}.reveal.linear section{-webkit-backface-visibility:hidden;backface-visibility:hidden}.reveal .slides>section[data-transition=linear].past,.reveal .slides>section[data-transition~=linear-out].past,.reveal.linear .slides>section:not([data-transition]).past{transform:translate(-150%,0)}.reveal .slides>section[data-transition=linear].future,.reveal .slides>section[data-transition~=linear-in].future,.reveal.linear .slides>section:not([data-transition]).future{transform:translate(150%,0)}.reveal .slides>section>section[data-transition=linear].past,.reveal .slides>section>section[data-transition~=linear-out].past,.reveal.linear .slides>section>section:not([data-transition]).past{transform:translate(0,-150%)}.reveal .slides>section>section[data-transition=linear].future,.reveal .slides>section>section[data-transition~=linear-in].future,.reveal.linear .slides>section>section:not([data-transition]).future{transform:translate(0,150%)}.reveal .slides section[data-transition=default].stack,.reveal.default .slides section.stack{transform-style:preserve-3d}.reveal .slides>section[data-transition=default].past,.reveal .slides>section[data-transition~=default-out].past,.reveal.default .slides>section:not([data-transition]).past{transform:translate3d(-100%,0,0) rotateY(-90deg) translate3d(-100%,0,0)}.reveal .slides>section[data-transition=default].future,.reveal .slides>section[data-transition~=default-in].future,.reveal.default .slides>section:not([data-transition]).future{transform:translate3d(100%,0,0) rotateY(90deg) translate3d(100%,0,0)}.reveal .slides>section>section[data-transition=default].past,.reveal .slides>section>section[data-transition~=default-out].past,.reveal.default .slides>section>section:not([data-transition]).past{transform:translate3d(0,-300px,0) rotateX(70deg) translate3d(0,-300px,0)}.reveal .slides>section>section[data-transition=default].future,.reveal .slides>section>section[data-transition~=default-in].future,.reveal.default .slides>section>section:not([data-transition]).future{transform:translate3d(0,300px,0) rotateX(-70deg) translate3d(0,300px,0)}.reveal .slides section[data-transition=convex].stack,.reveal.convex .slides section.stack{transform-style:preserve-3d}.reveal .slides>section[data-transition=convex].past,.reveal .slides>section[data-transition~=convex-out].past,.reveal.convex .slides>section:not([data-transition]).past{transform:translate3d(-100%,0,0) rotateY(-90deg) translate3d(-100%,0,0)}.reveal .slides>section[data-transition=convex].future,.reveal .slides>section[data-transition~=convex-in].future,.reveal.convex .slides>section:not([data-transition]).future{transform:translate3d(100%,0,0) rotateY(90deg) translate3d(100%,0,0)}.reveal .slides>section>section[data-transition=convex].past,.reveal .slides>section>section[data-transition~=convex-out].past,.reveal.convex .slides>section>section:not([data-transition]).past{transform:translate3d(0,-300px,0) rotateX(70deg) translate3d(0,-300px,0)}.reveal .slides>section>section[data-transition=convex].future,.reveal .slides>section>section[data-transition~=convex-in].future,.reveal.convex .slides>section>section:not([data-transition]).future{transform:translate3d(0,300px,0) rotateX(-70deg) translate3d(0,300px,0)}.reveal .slides section[data-transition=concave].stack,.reveal.concave .slides section.stack{transform-style:preserve-3d}.reveal .slides>section[data-transition=concave].past,.reveal .slides>section[data-transition~=concave-out].past,.reveal.concave .slides>section:not([data-transition]).past{transform:translate3d(-100%,0,0) rotateY(90deg) translate3d(-100%,0,0)}.reveal .slides>section[data-transition=concave].future,.reveal .slides>section[data-transition~=concave-in].future,.reveal.concave .slides>section:not([data-transition]).future{transform:translate3d(100%,0,0) rotateY(-90deg) translate3d(100%,0,0)}.reveal .slides>section>section[data-transition=concave].past,.reveal .slides>section>section[data-transition~=concave-out].past,.reveal.concave .slides>section>section:not([data-transition]).past{transform:translate3d(0,-80%,0) rotateX(-70deg) translate3d(0,-80%,0)}.reveal .slides>section>section[data-transition=concave].future,.reveal .slides>section>section[data-transition~=concave-in].future,.reveal.concave .slides>section>section:not([data-transition]).future{transform:translate3d(0,80%,0) rotateX(70deg) translate3d(0,80%,0)}.reveal .slides section[data-transition=zoom],.reveal.zoom .slides section:not([data-transition]){transition-timing-function:ease}.reveal .slides>section[data-transition=zoom].past,.reveal .slides>section[data-transition~=zoom-out].past,.reveal.zoom .slides>section:not([data-transition]).past{visibility:hidden;transform:scale(16)}.reveal .slides>section[data-transition=zoom].future,.reveal .slides>section[data-transition~=zoom-in].future,.reveal.zoom .slides>section:not([data-transition]).future{visibility:hidden;transform:scale(.2)}.reveal .slides>section>section[data-transition=zoom].past,.reveal .slides>section>section[data-transition~=zoom-out].past,.reveal.zoom .slides>section>section:not([data-transition]).past{transform:scale(16)}.reveal .slides>section>section[data-transition=zoom].future,.reveal .slides>section>section[data-transition~=zoom-in].future,.reveal.zoom .slides>section>section:not([data-transition]).future{transform:scale(.2)}.reveal.cube .slides{perspective:1300px}.reveal.cube .slides section{padding:30px;min-height:700px;-webkit-backface-visibility:hidden;backface-visibility:hidden;box-sizing:border-box;transform-style:preserve-3d}.reveal.center.cube .slides section{min-height:0}.reveal.cube .slides section:not(.stack):before{content:'';position:absolute;display:block;width:100%;height:100%;left:0;top:0;background:rgba(0,0,0,.1);border-radius:4px;transform:translateZ(-20px)}.reveal.cube .slides section:not(.stack):after{content:'';position:absolute;display:block;width:90%;height:30px;left:5%;bottom:0;background:0 0;z-index:1;border-radius:4px;box-shadow:0 95px 25px rgba(0,0,0,.2);transform:translateZ(-90px) rotateX(65deg)}.reveal.cube .slides>section.stack{padding:0;background:0 0}.reveal.cube .slides>section.past{transform-origin:100% 0;transform:translate3d(-100%,0,0) rotateY(-90deg)}.reveal.cube .slides>section.future{transform-origin:0 0;transform:translate3d(100%,0,0) rotateY(90deg)}.reveal.cube .slides>section>section.past{transform-origin:0 100%;transform:translate3d(0,-100%,0) rotateX(90deg)}.reveal.cube .slides>section>section.future{transform-origin:0 0;transform:translate3d(0,100%,0) rotateX(-90deg)}.reveal.page .slides{perspective-origin:0 50%;perspective:3000px}.reveal.page .slides section{padding:30px;min-height:700px;box-sizing:border-box;transform-style:preserve-3d}.reveal.page .slides section.past{z-index:12}.reveal.page .slides section:not(.stack):before{content:'';position:absolute;display:block;width:100%;height:100%;left:0;top:0;background:rgba(0,0,0,.1);transform:translateZ(-20px)}.reveal.page .slides section:not(.stack):after{content:'';position:absolute;display:block;width:90%;height:30px;left:5%;bottom:0;background:0 0;z-index:1;border-radius:4px;box-shadow:0 95px 25px rgba(0,0,0,.2);-webkit-transform:translateZ(-90px) rotateX(65deg)}.reveal.page .slides>section.stack{padding:0;background:0 0}.reveal.page .slides>section.past{transform-origin:0 0;transform:translate3d(-40%,0,0) rotateY(-80deg)}.reveal.page .slides>section.future{transform-origin:100% 0;transform:translate3d(0,0,0)}.reveal.page .slides>section>section.past{transform-origin:0 0;transform:translate3d(0,-40%,0) rotateX(80deg)}.reveal.page .slides>section>section.future{transform-origin:0 100%;transform:translate3d(0,0,0)}.reveal .slides section[data-transition=fade],.reveal.fade .slides section:not([data-transition]),.reveal.fade .slides>section>section:not([data-transition]){transform:none;transition:opacity .5s}.reveal.fade.overview .slides section,.reveal.fade.overview .slides>section>section{transition:none}.reveal .slides section[data-transition=none],.reveal.none .slides section:not([data-transition]){transform:none;transition:none}.reveal .pause-overlay{position:absolute;top:0;left:0;width:100%;height:100%;background:#000;visibility:hidden;opacity:0;z-index:100;transition:all 1s ease}.reveal .pause-overlay .resume-button{position:absolute;bottom:20px;right:20px;color:#ccc;border-radius:2px;padding:6px 14px;border:2px solid #ccc;font-size:16px;background:0 0;cursor:pointer}.reveal .pause-overlay .resume-button:hover{color:#fff;border-color:#fff}.reveal.paused .pause-overlay{visibility:visible;opacity:1}.reveal .no-transition,.reveal .no-transition *,.reveal .slides.disable-slide-transitions section{transition:none!important}.reveal .slides.disable-slide-transitions section{transform:none!important}.reveal .backgrounds{position:absolute;width:100%;height:100%;top:0;left:0;perspective:600px}.reveal .slide-background{display:none;position:absolute;width:100%;height:100%;opacity:0;visibility:hidden;overflow:hidden;background-color:rgba(0,0,0,0);transition:all .8s cubic-bezier(.26,.86,.44,.985)}.reveal .slide-background-content{position:absolute;width:100%;height:100%;background-position:50% 50%;background-repeat:no-repeat;background-size:cover}.reveal .slide-background.stack{display:block}.reveal .slide-background.present{opacity:1;visibility:visible;z-index:2}.print-pdf .reveal .slide-background{opacity:1!important;visibility:visible!important}.reveal .slide-background video{position:absolute;width:100%;height:100%;max-width:none;max-height:none;top:0;left:0;-o-object-fit:cover;object-fit:cover}.reveal .slide-background[data-background-size=contain] video{-o-object-fit:contain;object-fit:contain}.reveal>.backgrounds .slide-background[data-background-transition=none],.reveal[data-background-transition=none]>.backgrounds .slide-background{transition:none}.reveal>.backgrounds .slide-background[data-background-transition=slide],.reveal[data-background-transition=slide]>.backgrounds .slide-background{opacity:1;-webkit-backface-visibility:hidden;backface-visibility:hidden}.reveal>.backgrounds .slide-background.past[data-background-transition=slide],.reveal[data-background-transition=slide]>.backgrounds .slide-background.past{transform:translate(-100%,0)}.reveal>.backgrounds .slide-background.future[data-background-transition=slide],.reveal[data-background-transition=slide]>.backgrounds .slide-background.future{transform:translate(100%,0)}.reveal>.backgrounds .slide-background>.slide-background.past[data-background-transition=slide],.reveal[data-background-transition=slide]>.backgrounds .slide-background>.slide-background.past{transform:translate(0,-100%)}.reveal>.backgrounds .slide-background>.slide-background.future[data-background-transition=slide],.reveal[data-background-transition=slide]>.backgrounds .slide-background>.slide-background.future{transform:translate(0,100%)}.reveal>.backgrounds .slide-background.past[data-background-transition=convex],.reveal[data-background-transition=convex]>.backgrounds .slide-background.past{opacity:0;transform:translate3d(-100%,0,0) rotateY(-90deg) translate3d(-100%,0,0)}.reveal>.backgrounds .slide-background.future[data-background-transition=convex],.reveal[data-background-transition=convex]>.backgrounds .slide-background.future{opacity:0;transform:translate3d(100%,0,0) rotateY(90deg) translate3d(100%,0,0)}.reveal>.backgrounds .slide-background>.slide-background.past[data-background-transition=convex],.reveal[data-background-transition=convex]>.backgrounds .slide-background>.slide-background.past{opacity:0;transform:translate3d(0,-100%,0) rotateX(90deg) translate3d(0,-100%,0)}.reveal>.backgrounds .slide-background>.slide-background.future[data-background-transition=convex],.reveal[data-background-transition=convex]>.backgrounds .slide-background>.slide-background.future{opacity:0;transform:translate3d(0,100%,0) rotateX(-90deg) translate3d(0,100%,0)}.reveal>.backgrounds .slide-background.past[data-background-transition=concave],.reveal[data-background-transition=concave]>.backgrounds .slide-background.past{opacity:0;transform:translate3d(-100%,0,0) rotateY(90deg) translate3d(-100%,0,0)}.reveal>.backgrounds .slide-background.future[data-background-transition=concave],.reveal[data-background-transition=concave]>.backgrounds .slide-background.future{opacity:0;transform:translate3d(100%,0,0) rotateY(-90deg) translate3d(100%,0,0)}.reveal>.backgrounds .slide-background>.slide-background.past[data-background-transition=concave],.reveal[data-background-transition=concave]>.backgrounds .slide-background>.slide-background.past{opacity:0;transform:translate3d(0,-100%,0) rotateX(-90deg) translate3d(0,-100%,0)}.reveal>.backgrounds .slide-background>.slide-background.future[data-background-transition=concave],.reveal[data-background-transition=concave]>.backgrounds .slide-background>.slide-background.future{opacity:0;transform:translate3d(0,100%,0) rotateX(90deg) translate3d(0,100%,0)}.reveal>.backgrounds .slide-background[data-background-transition=zoom],.reveal[data-background-transition=zoom]>.backgrounds .slide-background{transition-timing-function:ease}.reveal>.backgrounds .slide-background.past[data-background-transition=zoom],.reveal[data-background-transition=zoom]>.backgrounds .slide-background.past{opacity:0;visibility:hidden;transform:scale(16)}.reveal>.backgrounds .slide-background.future[data-background-transition=zoom],.reveal[data-background-transition=zoom]>.backgrounds .slide-background.future{opacity:0;visibility:hidden;transform:scale(.2)}.reveal>.backgrounds .slide-background>.slide-background.past[data-background-transition=zoom],.reveal[data-background-transition=zoom]>.backgrounds .slide-background>.slide-background.past{opacity:0;visibility:hidden;transform:scale(16)}.reveal>.backgrounds .slide-background>.slide-background.future[data-background-transition=zoom],.reveal[data-background-transition=zoom]>.backgrounds .slide-background>.slide-background.future{opacity:0;visibility:hidden;transform:scale(.2)}.reveal[data-transition-speed=fast]>.backgrounds .slide-background{transition-duration:.4s}.reveal[data-transition-speed=slow]>.backgrounds .slide-background{transition-duration:1.2s}.reveal [data-auto-animate-target^=unmatched]{will-change:opacity}.reveal section[data-auto-animate]:not(.stack):not([data-auto-animate=running]) [data-auto-animate-target^=unmatched]{opacity:0}.reveal.overview{perspective-origin:50% 50%;perspective:700px}.reveal.overview .slides{-moz-transform-style:preserve-3d}.reveal.overview .slides section{height:100%;top:0!important;opacity:1!important;overflow:hidden;visibility:visible!important;cursor:pointer;box-sizing:border-box}.reveal.overview .slides section.present,.reveal.overview .slides section:hover{outline:10px solid rgba(150,150,150,.4);outline-offset:10px}.reveal.overview .slides section .fragment{opacity:1;transition:none}.reveal.overview .slides section:after,.reveal.overview .slides section:before{display:none!important}.reveal.overview .slides>section.stack{padding:0;top:0!important;background:0 0;outline:0;overflow:visible}.reveal.overview .backgrounds{perspective:inherit;-moz-transform-style:preserve-3d}.reveal.overview .backgrounds .slide-background{opacity:1;visibility:visible;outline:10px solid rgba(150,150,150,.1);outline-offset:10px}.reveal.overview .backgrounds .slide-background.stack{overflow:visible}.reveal.overview .slides section,.reveal.overview-deactivating .slides section{transition:none}.reveal.overview .backgrounds .slide-background,.reveal.overview-deactivating .backgrounds .slide-background{transition:none}.reveal.rtl .slides,.reveal.rtl .slides h1,.reveal.rtl .slides h2,.reveal.rtl .slides h3,.reveal.rtl .slides h4,.reveal.rtl .slides h5,.reveal.rtl .slides h6{direction:rtl;font-family:sans-serif}.reveal.rtl code,.reveal.rtl pre{direction:ltr}.reveal.rtl ol,.reveal.rtl ul{text-align:right}.reveal.rtl .progress span{transform-origin:100% 0}.reveal.has-parallax-background .backgrounds{transition:all .8s ease}.reveal.has-parallax-background[data-transition-speed=fast] .backgrounds{transition-duration:.4s}.reveal.has-parallax-background[data-transition-speed=slow] .backgrounds{transition-duration:1.2s}.reveal>.overlay{position:absolute;top:0;left:0;width:100%;height:100%;z-index:1000;background:rgba(0,0,0,.9);transition:all .3s ease}.reveal>.overlay .spinner{position:absolute;display:block;top:50%;left:50%;width:32px;height:32px;margin:-16px 0 0 -16px;z-index:10;background-image:url(%2F%2F%2F6%2Bvr8nJybW1tcDAwOjo6Nvb26ioqKOjo7Ozs%2FLy8vz8%2FAAAAAAAAAAAACH%2FC05FVFNDQVBFMi4wAwEAAAAh%2FhpDcmVhdGVkIHdpdGggYWpheGxvYWQuaW5mbwAh%2BQQJCgAAACwAAAAAIAAgAAAE5xDISWlhperN52JLhSSdRgwVo1ICQZRUsiwHpTJT4iowNS8vyW2icCF6k8HMMBkCEDskxTBDAZwuAkkqIfxIQyhBQBFvAQSDITM5VDW6XNE4KagNh6Bgwe60smQUB3d4Rz1ZBApnFASDd0hihh12BkE9kjAJVlycXIg7CQIFA6SlnJ87paqbSKiKoqusnbMdmDC2tXQlkUhziYtyWTxIfy6BE8WJt5YJvpJivxNaGmLHT0VnOgSYf0dZXS7APdpB309RnHOG5gDqXGLDaC457D1zZ%2FV%2FnmOM82XiHRLYKhKP1oZmADdEAAAh%2BQQJCgAAACwAAAAAIAAgAAAE6hDISWlZpOrNp1lGNRSdRpDUolIGw5RUYhhHukqFu8DsrEyqnWThGvAmhVlteBvojpTDDBUEIFwMFBRAmBkSgOrBFZogCASwBDEY%2FCZSg7GSE0gSCjQBMVG023xWBhklAnoEdhQEfyNqMIcKjhRsjEdnezB%2BA4k8gTwJhFuiW4dokXiloUepBAp5qaKpp6%2BHo7aWW54wl7obvEe0kRuoplCGepwSx2jJvqHEmGt6whJpGpfJCHmOoNHKaHx61WiSR92E4lbFoq%2BB6QDtuetcaBPnW6%2BO7wDHpIiK9SaVK5GgV543tzjgGcghAgAh%2BQQJCgAAACwAAAAAIAAgAAAE7hDISSkxpOrN5zFHNWRdhSiVoVLHspRUMoyUakyEe8PTPCATW9A14E0UvuAKMNAZKYUZCiBMuBakSQKG8G2FzUWox2AUtAQFcBKlVQoLgQReZhQlCIJesQXI5B0CBnUMOxMCenoCfTCEWBsJColTMANldx15BGs8B5wlCZ9Po6OJkwmRpnqkqnuSrayqfKmqpLajoiW5HJq7FL1Gr2mMMcKUMIiJgIemy7xZtJsTmsM4xHiKv5KMCXqfyUCJEonXPN2rAOIAmsfB3uPoAK%2B%2BG%2Bw48edZPK%2BM6hLJpQg484enXIdQFSS1u6UhksENEQAAIfkECQoAAAAsAAAAACAAIAAABOcQyEmpGKLqzWcZRVUQnZYg1aBSh2GUVEIQ2aQOE%2BG%2BcD4ntpWkZQj1JIiZIogDFFyHI0UxQwFugMSOFIPJftfVAEoZLBbcLEFhlQiqGp1Vd140AUklUN3eCA51C1EWMzMCezCBBmkxVIVHBWd3HHl9JQOIJSdSnJ0TDKChCwUJjoWMPaGqDKannasMo6WnM562R5YluZRwur0wpgqZE7NKUm%2BFNRPIhjBJxKZteWuIBMN4zRMIVIhffcgojwCF117i4nlLnY5ztRLsnOk%2BaV%2BoJY7V7m76PdkS4trKcdg0Zc0tTcKkRAAAIfkECQoAAAAsAAAAACAAIAAABO4QyEkpKqjqzScpRaVkXZWQEximw1BSCUEIlDohrft6cpKCk5xid5MNJTaAIkekKGQkWyKHkvhKsR7ARmitkAYDYRIbUQRQjWBwJRzChi9CRlBcY1UN4g0%2FVNB0AlcvcAYHRyZPdEQFYV8ccwR5HWxEJ02YmRMLnJ1xCYp0Y5idpQuhopmmC2KgojKasUQDk5BNAwwMOh2RtRq5uQuPZKGIJQIGwAwGf6I0JXMpC8C7kXWDBINFMxS4DKMAWVWAGYsAdNqW5uaRxkSKJOZKaU3tPOBZ4DuK2LATgJhkPJMgTwKCdFjyPHEnKxFCDhEAACH5BAkKAAAALAAAAAAgACAAAATzEMhJaVKp6s2nIkolIJ2WkBShpkVRWqqQrhLSEu9MZJKK9y1ZrqYK9WiClmvoUaF8gIQSNeF1Er4MNFn4SRSDARWroAIETg1iVwuHjYB1kYc1mwruwXKC9gmsJXliGxc%2BXiUCby9ydh1sOSdMkpMTBpaXBzsfhoc5l58Gm5yToAaZhaOUqjkDgCWNHAULCwOLaTmzswadEqggQwgHuQsHIoZCHQMMQgQGubVEcxOPFAcMDAYUA85eWARmfSRQCdcMe0zeP1AAygwLlJtPNAAL19DARdPzBOWSm1brJBi45soRAWQAAkrQIykShQ9wVhHCwCQCACH5BAkKAAAALAAAAAAgACAAAATrEMhJaVKp6s2nIkqFZF2VIBWhUsJaTokqUCoBq%2BE71SRQeyqUToLA7VxF0JDyIQh%2FMVVPMt1ECZlfcjZJ9mIKoaTl1MRIl5o4CUKXOwmyrCInCKqcWtvadL2SYhyASyNDJ0uIiRMDjI0Fd30%2FiI2UA5GSS5UDj2l6NoqgOgN4gksEBgYFf0FDqKgHnyZ9OX8HrgYHdHpcHQULXAS2qKpENRg7eAMLC7kTBaixUYFkKAzWAAnLC7FLVxLWDBLKCwaKTULgEwbLA4hJtOkSBNqITT3xEgfLpBtzE%2FjiuL04RGEBgwWhShRgQExHBAAh%2BQQJCgAAACwAAAAAIAAgAAAE7xDISWlSqerNpyJKhWRdlSAVoVLCWk6JKlAqAavhO9UkUHsqlE6CwO1cRdCQ8iEIfzFVTzLdRAmZX3I2SfZiCqGk5dTESJeaOAlClzsJsqwiJwiqnFrb2nS9kmIcgEsjQydLiIlHehhpejaIjzh9eomSjZR%2BipslWIRLAgMDOR2DOqKogTB9pCUJBagDBXR6XB0EBkIIsaRsGGMMAxoDBgYHTKJiUYEGDAzHC9EACcUGkIgFzgwZ0QsSBcXHiQvOwgDdEwfFs0sDzt4S6BK4xYjkDOzn0unFeBzOBijIm1Dgmg5YFQwsCMjp1oJ8LyIAACH5BAkKAAAALAAAAAAgACAAAATwEMhJaVKp6s2nIkqFZF2VIBWhUsJaTokqUCoBq%2BE71SRQeyqUToLA7VxF0JDyIQh%2FMVVPMt1ECZlfcjZJ9mIKoaTl1MRIl5o4CUKXOwmyrCInCKqcWtvadL2SYhyASyNDJ0uIiUd6GGl6NoiPOH16iZKNlH6KmyWFOggHhEEvAwwMA0N9GBsEC6amhnVcEwavDAazGwIDaH1ipaYLBUTCGgQDA8NdHz0FpqgTBwsLqAbWAAnIA4FWKdMLGdYGEgraigbT0OITBcg5QwPT4xLrROZL6AuQAPUS7bxLpoWidY0JtxLHKhwwMJBTHgPKdEQAACH5BAkKAAAALAAAAAAgACAAAATrEMhJaVKp6s2nIkqFZF2VIBWhUsJaTokqUCoBq%2BE71SRQeyqUToLA7VxF0JDyIQh%2FMVVPMt1ECZlfcjZJ9mIKoaTl1MRIl5o4CUKXOwmyrCInCKqcWtvadL2SYhyASyNDJ0uIiUd6GAULDJCRiXo1CpGXDJOUjY%2BYip9DhToJA4RBLwMLCwVDfRgbBAaqqoZ1XBMHswsHtxtFaH1iqaoGNgAIxRpbFAgfPQSqpbgGBqUD1wBXeCYp1AYZ19JJOYgH1KwA4UBvQwXUBxPqVD9L3sbp2BNk2xvvFPJd%2BMFCN6HAAIKgNggY0KtEBAAh%2BQQJCgAAACwAAAAAIAAgAAAE6BDISWlSqerNpyJKhWRdlSAVoVLCWk6JKlAqAavhO9UkUHsqlE6CwO1cRdCQ8iEIfzFVTzLdRAmZX3I2SfYIDMaAFdTESJeaEDAIMxYFqrOUaNW4E4ObYcCXaiBVEgULe0NJaxxtYksjh2NLkZISgDgJhHthkpU4mW6blRiYmZOlh4JWkDqILwUGBnE6TYEbCgevr0N1gH4At7gHiRpFaLNrrq8HNgAJA70AWxQIH1%2BvsYMDAzZQPC9VCNkDWUhGkuE5PxJNwiUK4UfLzOlD4WvzAHaoG9nxPi5d%2BjYUqfAhhykOFwJWiAAAIfkECQoAAAAsAAAAACAAIAAABPAQyElpUqnqzaciSoVkXVUMFaFSwlpOCcMYlErAavhOMnNLNo8KsZsMZItJEIDIFSkLGQoQTNhIsFehRww2CQLKF0tYGKYSg%2BygsZIuNqJksKgbfgIGepNo2cIUB3V1B3IvNiBYNQaDSTtfhhx0CwVPI0UJe0%2Bbm4g5VgcGoqOcnjmjqDSdnhgEoamcsZuXO1aWQy8KAwOAuTYYGwi7w5h%2BKr0SJ8MFihpNbx%2B4Erq7BYBuzsdiH1jCAzoSfl0rVirNbRXlBBlLX%2BBP0XJLAPGzTkAuAOqb0WT5AH7OcdCm5B8TgRwSRKIHQtaLCwg1RAAAOwAAAAAAAAAAAA%3D%3D);visibility:visible;opacity:.6;transition:all .3s ease}.reveal>.overlay header{position:absolute;left:0;top:0;width:100%;padding:5px;z-index:2;box-sizing:border-box}.reveal>.overlay header a{display:inline-block;width:40px;height:40px;line-height:36px;padding:0 10px;float:right;opacity:.6;box-sizing:border-box}.reveal>.overlay header a:hover{opacity:1}.reveal>.overlay header a .icon{display:inline-block;width:20px;height:20px;background-position:50% 50%;background-size:100%;background-repeat:no-repeat}.reveal>.overlay header a.close .icon{background-image:url()}.reveal>.overlay header a.external .icon{background-image:url()}.reveal>.overlay .viewport{position:absolute;display:flex;top:50px;right:0;bottom:0;left:0}.reveal>.overlay.overlay-preview .viewport iframe{width:100%;height:100%;max-width:100%;max-height:100%;border:0;opacity:0;visibility:hidden;transition:all .3s ease}.reveal>.overlay.overlay-preview.loaded .viewport iframe{opacity:1;visibility:visible}.reveal>.overlay.overlay-preview.loaded .viewport-inner{position:absolute;z-index:-1;left:0;top:45%;width:100%;text-align:center;letter-spacing:normal}.reveal>.overlay.overlay-preview .x-frame-error{opacity:0;transition:opacity .3s ease .3s}.reveal>.overlay.overlay-preview.loaded .x-frame-error{opacity:1}.reveal>.overlay.overlay-preview.loaded .spinner{opacity:0;visibility:hidden;transform:scale(.2)}.reveal>.overlay.overlay-help .viewport{overflow:auto;color:#fff}.reveal>.overlay.overlay-help .viewport .viewport-inner{width:600px;margin:auto;padding:20px 20px 80px 20px;text-align:center;letter-spacing:normal}.reveal>.overlay.overlay-help .viewport .viewport-inner .title{font-size:20px}.reveal>.overlay.overlay-help .viewport .viewport-inner table{border:1px solid #fff;border-collapse:collapse;font-size:16px}.reveal>.overlay.overlay-help .viewport .viewport-inner table td,.reveal>.overlay.overlay-help .viewport .viewport-inner table th{width:200px;padding:14px;border:1px solid #fff;vertical-align:middle}.reveal>.overlay.overlay-help .viewport .viewport-inner table th{padding-top:20px;padding-bottom:20px}.reveal .playback{position:absolute;left:15px;bottom:20px;z-index:30;cursor:pointer;transition:all .4s ease;-webkit-tap-highlight-color:transparent}.reveal.overview .playback{opacity:0;visibility:hidden}.reveal .hljs{min-height:100%}.reveal .hljs table{margin:initial}.reveal .hljs-ln-code,.reveal .hljs-ln-numbers{padding:0;border:0}.reveal .hljs-ln-numbers{opacity:.6;padding-right:.75em;text-align:right;vertical-align:top}.reveal .hljs.has-highlights tr:not(.highlight-line){opacity:.4}.reveal .hljs:not(:first-child).fragment{position:absolute;top:0;left:0;width:100%;box-sizing:border-box}.reveal pre[data-auto-animate-target]{overflow:hidden}.reveal pre[data-auto-animate-target] code{height:100%}.reveal .roll{display:inline-block;line-height:1.2;overflow:hidden;vertical-align:top;perspective:400px;perspective-origin:50% 50%}.reveal .roll:hover{background:0 0;text-shadow:none}.reveal .roll span{display:block;position:relative;padding:0 2px;pointer-events:none;transition:all .4s ease;transform-origin:50% 0;transform-style:preserve-3d;-webkit-backface-visibility:hidden;backface-visibility:hidden}.reveal .roll:hover span{background:rgba(0,0,0,.5);transform:translate3d(0,0,-45px) rotateX(90deg)}.reveal .roll span:after{content:attr(data-title);display:block;position:absolute;left:0;top:0;padding:0 2px;-webkit-backface-visibility:hidden;backface-visibility:hidden;transform-origin:50% 0;transform:translate3d(0,110%,0) rotateX(-90deg)}.reveal aside.notes{display:none}.reveal .speaker-notes{display:none;position:absolute;width:33.33333%;height:100%;top:0;left:100%;padding:14px 18px 14px 18px;z-index:1;font-size:18px;line-height:1.4;border:1px solid rgba(0,0,0,.05);color:#222;background-color:#f5f5f5;overflow:auto;box-sizing:border-box;text-align:left;font-family:Helvetica,sans-serif;-webkit-overflow-scrolling:touch}.reveal .speaker-notes .notes-placeholder{color:#ccc;font-style:italic}.reveal .speaker-notes:focus{outline:0}.reveal .speaker-notes:before{content:'Speaker notes';display:block;margin-bottom:10px;opacity:.5}.reveal.show-notes{max-width:75%;overflow:visible}.reveal.show-notes .speaker-notes{display:block}@media screen and (min-width:1600px){.reveal .speaker-notes{font-size:20px}}@media screen and (max-width:1024px){.reveal.show-notes{border-left:0;max-width:none;max-height:70%;max-height:70vh;overflow:visible}.reveal.show-notes .speaker-notes{top:100%;left:0;width:100%;height:42.85714%;height:30vh;border:0}}@media screen and (max-width:600px){.reveal.show-notes{max-height:60%;max-height:60vh}.reveal.show-notes .speaker-notes{top:100%;height:66.66667%;height:40vh}.reveal .speaker-notes{font-size:14px}}.zoomed .reveal *,.zoomed .reveal :after,.zoomed .reveal :before{-webkit-backface-visibility:visible!important;backface-visibility:visible!important}.zoomed .reveal .controls,.zoomed .reveal .progress{opacity:0}.zoomed .reveal .roll span{background:0 0}.zoomed .reveal .roll span:after{visibility:hidden}html.print-pdf *{-webkit-print-color-adjust:exact}html.print-pdf{width:100%;height:100%;overflow:visible}html.print-pdf body{margin:0 auto!important;border:0;padding:0;float:none!important;overflow:visible}html.print-pdf .nestedarrow,html.print-pdf .reveal .controls,html.print-pdf .reveal .playback,html.print-pdf .reveal .progress,html.print-pdf .reveal.overview,html.print-pdf .state-background{display:none!important}html.print-pdf .reveal pre code{overflow:hidden!important;font-family:Courier,'Courier New',monospace!important}html.print-pdf .reveal{width:auto!important;height:auto!important;overflow:hidden!important}html.print-pdf .reveal .slides{position:static;width:100%!important;height:auto!important;zoom:1!important;pointer-events:initial;left:auto;top:auto;margin:0!important;padding:0!important;overflow:visible;display:block;perspective:none;perspective-origin:50% 50%}html.print-pdf .reveal .slides .pdf-page{position:relative;overflow:hidden;z-index:1;page-break-after:always}html.print-pdf .reveal .slides section{visibility:visible!important;display:block!important;position:absolute!important;margin:0!important;padding:0!important;box-sizing:border-box!important;min-height:1px;opacity:1!important;transform-style:flat!important;transform:none!important}html.print-pdf .reveal section.stack{position:relative!important;margin:0!important;padding:0!important;page-break-after:avoid!important;height:auto!important;min-height:auto!important}html.print-pdf .reveal img{box-shadow:none}html.print-pdf .reveal .backgrounds{display:none}html.print-pdf .reveal .slide-background{display:block!important;position:absolute;top:0;left:0;width:100%;height:100%;z-index:auto!important}html.print-pdf .reveal.show-notes{max-width:none;max-height:none}html.print-pdf .reveal .speaker-notes-pdf{display:block;width:100%;height:auto;max-height:none;top:auto;right:auto;bottom:auto;left:auto;z-index:100}html.print-pdf .reveal .speaker-notes-pdf[data-layout=separate-page]{position:relative;color:inherit;background-color:transparent;padding:20px;page-break-after:always;border:0}html.print-pdf .reveal .slide-number-pdf{display:block;position:absolute;font-size:14px}html.print-pdf .aria-status{display:none}@media print{html:not(.print-pdf){background:#fff;width:auto;height:auto;overflow:visible}html:not(.print-pdf) body{background:#fff;font-size:20pt;width:auto;height:auto;border:0;margin:0 5%;padding:0;overflow:visible;float:none!important}html:not(.print-pdf) .controls,html:not(.print-pdf) .fork-reveal,html:not(.print-pdf) .nestedarrow,html:not(.print-pdf) .reveal .backgrounds,html:not(.print-pdf) .reveal .progress,html:not(.print-pdf) .reveal .slide-number,html:not(.print-pdf) .share-reveal,html:not(.print-pdf) .state-background{display:none!important}html:not(.print-pdf) body,html:not(.print-pdf) li,html:not(.print-pdf) p,html:not(.print-pdf) td{font-size:20pt!important;color:#000}html:not(.print-pdf) h1,html:not(.print-pdf) h2,html:not(.print-pdf) h3,html:not(.print-pdf) h4,html:not(.print-pdf) h5,html:not(.print-pdf) h6{color:#000!important;height:auto;line-height:normal;text-align:left;letter-spacing:normal}html:not(.print-pdf) h1{font-size:28pt!important}html:not(.print-pdf) h2{font-size:24pt!important}html:not(.print-pdf) h3{font-size:22pt!important}html:not(.print-pdf) h4{font-size:22pt!important;font-variant:small-caps}html:not(.print-pdf) h5{font-size:21pt!important}html:not(.print-pdf) h6{font-size:20pt!important;font-style:italic}html:not(.print-pdf) a:link,html:not(.print-pdf) a:visited{color:#000!important;font-weight:700;text-decoration:underline}html:not(.print-pdf) div,html:not(.print-pdf) ol,html:not(.print-pdf) p,html:not(.print-pdf) ul{visibility:visible;position:static;width:auto;height:auto;display:block;overflow:visible;margin:0;text-align:left!important}html:not(.print-pdf) .reveal pre,html:not(.print-pdf) .reveal table{margin-left:0;margin-right:0}html:not(.print-pdf) .reveal pre code{padding:20px}html:not(.print-pdf) .reveal blockquote{margin:20px 0}html:not(.print-pdf) .reveal .slides{position:static!important;width:auto!important;height:auto!important;left:0!important;top:0!important;margin-left:0!important;margin-top:0!important;padding:0!important;zoom:1!important;transform:none!important;overflow:visible!important;display:block!important;text-align:left!important;perspective:none;perspective-origin:50% 50%}html:not(.print-pdf) .reveal .slides section{visibility:visible!important;position:static!important;width:auto!important;height:auto!important;display:block!important;overflow:visible!important;left:0!important;top:0!important;margin-left:0!important;margin-top:0!important;padding:60px 20px!important;z-index:auto!important;opacity:1!important;page-break-after:always!important;transform-style:flat!important;transform:none!important;transition:none!important}html:not(.print-pdf) .reveal .slides section.stack{padding:0!important}html:not(.print-pdf) .reveal section:last-of-type{page-break-after:avoid!important}html:not(.print-pdf) .reveal section .fragment{opacity:1!important;visibility:visible!important;transform:none!important}html:not(.print-pdf) .reveal section img{display:block;margin:15px 0;background:#fff;border:1px solid #666;box-shadow:none}html:not(.print-pdf) .reveal section small{font-size:.8em}html:not(.print-pdf) .reveal .hljs{max-height:100%;white-space:pre-wrap;word-wrap:break-word;word-break:break-word;font-size:15pt}html:not(.print-pdf) .reveal .hljs .hljs-ln-numbers{white-space:nowrap}html:not(.print-pdf) .reveal .hljs td{font-size:inherit!important;color:inherit!important}}
      
      </style>
      
        <script type="text/x-mathjax-config">
          MathJax.Hub.Config({"extensions":["tex2jax.js"],"jax":["input/TeX","output/HTML-CSS"],"messageStyle":"none","tex2jax":{"processEnvironments":false,"processEscapes":true,"inlineMath":[["$","$"],["\\(","\\)"]],"displayMath":[["$$","$$"],["\\[","\\]"]],"skipTags":["script","noscript","style","textarea","pre","code"]},"displayAlign":"left","displayIndent":"0.05rem","TeX":{"equationNumbers":{"autoNumber":"none","useLabelIds":true},"extensions":["AMSmath.js","AMSsymbols.js","noErrors.js","noUndefined.js","action.js","cancel.js","enclose.js","mhchem.js","extpfeil.js"],"Macros":{"zerov":"{\\boldsymbol 0}","onev":"{\\boldsymbol 1}","av":"{\\boldsymbol a}","bv":"{\\boldsymbol b}","cv":"{\\boldsymbol c}","dv":"{\\boldsymbol d}","ev":"{\\boldsymbol e}","fv":"{\\boldsymbol f}","gv":"{\\boldsymbol g}","hv":"{\\boldsymbol h}","iv":"{\\boldsymbol i}","jv":"{\\boldsymbol j}","kv":"{\\boldsymbol k}","lv":"{\\boldsymbol l}","mv":"{\\boldsymbol m}","nv":"{\\boldsymbol n}","ov":"{\\boldsymbol o}","pv":"{\\boldsymbol p}","qv":"{\\boldsymbol q}","rv":"{\\boldsymbol r}","sv":"{\\boldsymbol s}","tv":"{\\boldsymbol t}","uv":"{\\boldsymbol u}","vv":"{\\boldsymbol v}","wv":"{\\boldsymbol w}","xv":"{\\boldsymbol x}","yv":"{\\boldsymbol y}","zv":"{\\boldsymbol z}","Av":"{\\mathbf A}","Bv":"{\\mathbf B}","Cv":"{\\mathbf C}","Dv":"{\\mathbf D}","Ev":"{\\mathbf E}","Fv":"{\\mathbf F}","Gv":"{\\mathbf G}","Hv":"{\\mathbf H}","Iv":"{\\mathbf I}","Jv":"{\\mathbf J}","Kv":"{\\mathbf K}","Lv":"{\\mathbf L}","Mv":"{\\mathbf M}","Nv":"{\\mathbf N}","Ov":"{\\mathbf O}","Pv":"{\\mathbf P}","Qv":"{\\mathbf Q}","Rv":"{\\mathbf R}","Sv":"{\\mathbf S}","Tv":"{\\mathbf T}","Uv":"{\\mathbf U}","Vv":"{\\mathbf V}","Wv":"{\\mathbf W}","Xv":"{\\mathbf X}","Yv":"{\\mathbf Y}","Zv":"{\\mathbf Z}","alphav":"{\\boldsymbol {\\alpha}}","betav":"{\\boldsymbol {\\beta}}","lambdav":"{\\boldsymbol {\\lambda}}","muv":"{\\boldsymbol {\\mu}}","thetav":"{\\boldsymbol {\\theta}}","phiv":"{\\boldsymbol {\\phi}}","zetav":"{\\boldsymbol {\\zeta}}","deltav":"{\\boldsymbol {\\delta}}","Sigmav":"{\\boldsymbol {\\Sigma}}","Phiv":"{\\boldsymbol {\\Phi}}","Lambdav":"{\\boldsymbol {\\Lambda}}","Omegav":"{\\boldsymbol {\\Omega}}","Cbb":"{\\mathbb C}","Ebb":"{\\mathbb E}","Hbb":"{\\mathbb H}","Nbb":"{\\mathbb N}","Pbb":"{\\mathbb P}","Qbb":"{\\mathbb Q}","Rbb":"{\\mathbb R}","Zbb":"{\\mathbb Z}","Acal":"{\\mathcal A}","Bcal":"{\\mathcal B}","Ccal":"{\\mathcal C}","Dcal":"{\\mathcal D}","Ecal":"{\\mathcal E}","Fcal":"{\\mathcal F}","Gcal":"{\\mathcal G}","Hcal":"{\\mathcal H}","Ical":"{\\mathcal I}","Lcal":"{\\mathcal L}","Mcal":"{\\mathcal M}","Ncal":"{\\mathcal N}","Pcal":"{\\mathcal P}","Rcal":"{\\mathcal R}","Scal":"{\\mathcal S}","Ucal":"{\\mathcal U}","Vcal":"{\\mathcal V}","Wcal":"{\\mathcal W}","Xcal":"{\\mathcal X}","Ycal":"{\\mathcal Y}","fhat":"{\\hat f}","yhat":"{\\hat y}","yvhat":"{\\hat {\\yv}}","Xvhat":"{\\hat {\\Xv}}","wvt":"{\\tilde {\\wv}}","xvt":"{\\tilde {\\xv}}","yvt":"{\\tilde {\\yv}}","Kvt":"{\\tilde {\\Kv}}","xbar":"{\\bar {x}}","ybar":"{\\bar {y}}","yvbar":"{\\bar {\\yv}}","Ffrak":"{\\mathfrak F}","sup":["{{(#1)}}",1],"diff":"{\\mathrm {d}}","diag":"{\\mathrm {diag}}","span":"{\\mathrm {span}}","sign":"{\\mathrm {sign}}","sgn":"{\\mathrm {sgn}}","st":"{\\mathrm {s.t.}}","VC":"{\\mathrm {VC}}","Pr":"{\\mathrm {Pr}}","tanh":"{\\mathrm {Tanh}}","relu":"{\\mathrm {ReLU}}","lrelu":"{\\mathrm {LeakyReLU}}","prelu":"{\\mathrm {PReLU}}","elu":"{\\mathrm {ELU}}","softplus":"{\\mathrm {Softplus}}","swish":"{\\mathrm {Swish}}","maxout":"{\\mathrm {Maxout}}","const":"{\\mathrm {const}}","cov":"{\\mathrm {cov}}","grad":"{\\mathrm {grad}}","div":"{\\mathrm {div}}","var":"{\\mathrm {var}}","softmax":"{\\mathrm {Softmax}}","att":"{\\mathrm {att}}","cut":"{\\mathrm {cut}}","rcut":"{\\mathrm {RatioCut}}","ncut":"{\\mathrm {NCut}}","tr":"{\\mathrm {tr}}","vol":"{\\mathrm {vol}}","mlp":"{\\mathrm {MLP}}","update":"{\\mathrm {Update}}","aggregate":"{\\mathrm {Aggregate}}","self":"{\\mathrm {self}}","set":"{\\mathrm {set}}","neigh":"{\\mathrm {neigh}}","base":"{\\mathrm {base}}","NULL":"{\\mathrm {NULL}}","new":"{\\mathrm {new}}","gru":"{\\mathrm {GRU}}","lstm":"{\\mathrm {LSTM}}","edge":"{\\mathrm {edge}}","node":"{\\mathrm {node}}","graph":"{\\mathrm {graph}}","train":"{\\mathrm {train}}","dec":"{\\mathrm {Dec}}","sym":"{\\mathrm {sym}}","modd":"{\\mathrm {mod} ~ }","hp":"{\\mathrm {hp}}","gen":"{\\mathrm {gen}}","rot":"{\\mathbf {rot180}}","up":"{\\mathbf {up}}","argmin":"{\\mathop{\\mathrm{argmin}}}","argmax":"{\\mathop{\\mathrm{argmax}}}"}},"HTML-CSS":{"linebreaks":{"automatic":false},"scale":100,"styles":{".MathJax_Display":{"margin":"0.6rem auto 1rem 0 !important","border-radius":"0px !important","font-size":"1.8rem !important","color":"#d33682","text-align":"left !important"},".MathJax":{"margin-left":"0.2rem !important","margin-right":"0rem !important","border":"0px solid #ccc !important","color":"#d33682"}},"availableFonts":["TeX"]}});
        </script>
        <script type="text/javascript" async="" src="../common/js/mathjax/MathJax.js" charset="UTF-8"></script>
        
      
      <link rel="stylesheet" href="file:////home/avanti/.local/bin/mpe2html-deps/node_modules/@shd101wyy/mume/dependencies/font-awesome/css/font-awesome.min.css">
      
        <script src="../common/js/head.min.js"></script>
        <script src="../common/js/reveal.js"></script>
      
      
      
      
      
      
      <style>
      /**
 * prism.js Github theme based on GitHub's theme.
 * @author Sam Clarke
 */
code[class*="language-"],
pre[class*="language-"] {
  color: #333;
  background: none;
  font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace;
  text-align: left;
  white-space: pre;
  word-spacing: normal;
  word-break: normal;
  word-wrap: normal;
  line-height: 1.4;

  -moz-tab-size: 8;
  -o-tab-size: 8;
  tab-size: 8;

  -webkit-hyphens: none;
  -moz-hyphens: none;
  -ms-hyphens: none;
  hyphens: none;
}

/* Code blocks */
pre[class*="language-"] {
  padding: .8em;
  overflow: auto;
  /* border: 1px solid #ddd; */
  border-radius: 3px;
  /* background: #fff; */
  background: #f5f5f5;
}

/* Inline code */
:not(pre) > code[class*="language-"] {
  padding: .1em;
  border-radius: .3em;
  white-space: normal;
  background: #f5f5f5;
}

.token.comment,
.token.blockquote {
  color: #969896;
}

.token.cdata {
  color: #183691;
}

.token.doctype,
.token.punctuation,
.token.variable,
.token.macro.property {
  color: #333;
}

.token.operator,
.token.important,
.token.keyword,
.token.rule,
.token.builtin {
  color: #a71d5d;
}

.token.string,
.token.url,
.token.regex,
.token.attr-value {
  color: #183691;
}

.token.property,
.token.number,
.token.boolean,
.token.entity,
.token.atrule,
.token.constant,
.token.symbol,
.token.command,
.token.code {
  color: #0086b3;
}

.token.tag,
.token.selector,
.token.prolog {
  color: #63a35c;
}

.token.function,
.token.namespace,
.token.pseudo-element,
.token.class,
.token.class-name,
.token.pseudo-class,
.token.id,
.token.url-reference .token.variable,
.token.attr-name {
  color: #795da3;
}

.token.entity {
  cursor: help;
}

.token.title,
.token.title .token.punctuation {
  font-weight: bold;
  color: #1d3e81;
}

.token.list {
  color: #ed6a43;
}

.token.inserted {
  background-color: #eaffea;
  color: #55a532;
}

.token.deleted {
  background-color: #ffecec;
  color: #bd2c00;
}

.token.bold {
  font-weight: bold;
}

.token.italic {
  font-style: italic;
}


/* JSON */
.language-json .token.property {
  color: #183691;
}

.language-markup .token.tag .token.punctuation {
  color: #333;
}

/* CSS */
code.language-css,
.language-css .token.function {
  color: #0086b3;
}

/* YAML */
.language-yaml .token.atrule {
  color: #63a35c;
}

code.language-yaml {
  color: #183691;
}

/* Ruby */
.language-ruby .token.function {
  color: #333;
}

/* Markdown */
.language-markdown .token.url {
  color: #795da3;
}

/* Makefile */
.language-makefile .token.symbol {
  color: #795da3;
}

.language-makefile .token.variable {
  color: #183691;
}

.language-makefile .token.builtin {
  color: #0086b3;
}

/* Bash */
.language-bash .token.keyword {
  color: #0086b3;
}

/* highlight */
pre[data-line] {
  position: relative;
  padding: 1em 0 1em 3em;
}
pre[data-line] .line-highlight-wrapper {
  position: absolute;
  top: 0;
  left: 0;
  background-color: transparent;
  display: block;
  width: 100%;
}

pre[data-line] .line-highlight {
  position: absolute;
  left: 0;
  right: 0;
  padding: inherit 0;
  margin-top: 1em;
  background: hsla(24, 20%, 50%,.08);
  background: linear-gradient(to right, hsla(24, 20%, 50%,.1) 70%, hsla(24, 20%, 50%,0));
  pointer-events: none;
  line-height: inherit;
  white-space: pre;
}

pre[data-line] .line-highlight:before, 
pre[data-line] .line-highlight[data-end]:after {
  content: attr(data-start);
  position: absolute;
  top: .4em;
  left: .6em;
  min-width: 1em;
  padding: 0 .5em;
  background-color: hsla(24, 20%, 50%,.4);
  color: hsl(24, 20%, 95%);
  font: bold 65%/1.5 sans-serif;
  text-align: center;
  vertical-align: .3em;
  border-radius: 999px;
  text-shadow: none;
  box-shadow: 0 1px white;
}

pre[data-line] .line-highlight[data-end]:after {
  content: attr(data-end);
  top: auto;
  bottom: .4em;
}.markdown-preview{width:100%;height:100%;box-sizing:border-box}.markdown-preview .pagebreak,.markdown-preview .newpage{page-break-before:always}.markdown-preview pre.line-numbers{position:relative;padding-left:3.8em;counter-reset:linenumber}.markdown-preview pre.line-numbers>code{position:relative}.markdown-preview pre.line-numbers .line-numbers-rows{position:absolute;pointer-events:none;top:1em;font-size:100%;left:0;width:3em;letter-spacing:-1px;border-right:1px solid #999;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}.markdown-preview pre.line-numbers .line-numbers-rows>span{pointer-events:none;display:block;counter-increment:linenumber}.markdown-preview pre.line-numbers .line-numbers-rows>span:before{content:counter(linenumber);color:#999;display:block;padding-right:.8em;text-align:right}.markdown-preview .mathjax-exps .MathJax_Display{text-align:center !important}.markdown-preview:not([for="preview"]) .code-chunk .btn-group{display:none}.markdown-preview:not([for="preview"]) .code-chunk .status{display:none}.markdown-preview:not([for="preview"]) .code-chunk .output-div{margin-bottom:16px}.scrollbar-style::-webkit-scrollbar{width:8px}.scrollbar-style::-webkit-scrollbar-track{border-radius:10px;background-color:transparent}.scrollbar-style::-webkit-scrollbar-thumb{border-radius:5px;background-color:rgba(150,150,150,0.66);border:4px solid rgba(150,150,150,0.66);background-clip:content-box}html body[for="html-export"]:not([data-presentation-mode]){position:relative;width:100%;height:100%;top:0;left:0;margin:0;padding:0;overflow:auto}html body[for="html-export"]:not([data-presentation-mode]) .markdown-preview{position:relative;top:0}@media screen and (min-width:914px){html body[for="html-export"]:not([data-presentation-mode]) .markdown-preview{padding:2em calc(50% - 457px + 2em)}}@media screen and (max-width:914px){html body[for="html-export"]:not([data-presentation-mode]) .markdown-preview{padding:2em}}@media screen and (max-width:450px){html body[for="html-export"]:not([data-presentation-mode]) .markdown-preview{font-size:14px !important;padding:1em}}@media print{html body[for="html-export"]:not([data-presentation-mode]) #sidebar-toc-btn{display:none}}html body[for="html-export"]:not([data-presentation-mode]) #sidebar-toc-btn{position:fixed;bottom:8px;left:8px;font-size:28px;cursor:pointer;color:inherit;z-index:99;width:32px;text-align:center;opacity:.4}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] #sidebar-toc-btn{opacity:1}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc{position:fixed;top:0;left:0;width:300px;height:100%;padding:32px 0 48px 0;font-size:14px;box-shadow:0 0 4px rgba(150,150,150,0.33);box-sizing:border-box;overflow:auto;background-color:inherit}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc::-webkit-scrollbar{width:8px}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc::-webkit-scrollbar-track{border-radius:10px;background-color:transparent}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc::-webkit-scrollbar-thumb{border-radius:5px;background-color:rgba(150,150,150,0.66);border:4px solid rgba(150,150,150,0.66);background-clip:content-box}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc a{text-decoration:none}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc ul{padding:0 1.6em;margin-top:.8em}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc li{margin-bottom:.8em}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc ul{list-style-type:none}html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .markdown-preview{left:300px;width:calc(100% -  300px);padding:2em calc(50% - 457px -  150px);margin:0;box-sizing:border-box}@media screen and (max-width:1274px){html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .markdown-preview{padding:2em}}@media screen and (max-width:450px){html body[for="html-export"]:not([data-presentation-mode])[html-show-sidebar-toc] .markdown-preview{width:100%}}html body[for="html-export"]:not([data-presentation-mode]):not([html-show-sidebar-toc]) .markdown-preview{left:50%;transform:translateX(-50%)}html body[for="html-export"]:not([data-presentation-mode]):not([html-show-sidebar-toc]) .md-sidebar-toc{display:none}
/* Please visit the URL below for more information: */
/*   https://shd101wyy.github.io/markdown-preview-enhanced/#/customize-css */

      </style>
    </head>
    <body for="html-export" data-presentation-mode="">
      <div class="mume markdown-preview  " data-presentation-mode="">
      
    <div style="display:none;"><link rel="stylesheet" href="../common/css/font-awesome-4.7.0/css/font-awesome.css">
<link rel="stylesheet" href="../common/css/style-color.css">
<link rel="stylesheet" href="../common/css/margin.css">
</div>
    <div class="reveal">
      <div class="slides">
        <section><section data-notes="" lineno="11" class="slide " data-line="11" data-h="0" data-v="0">
<div class="header"><img class="hust" src=""></div>
<div class="bottom15"></div>
<h1 class="mume-header" id="%E5%9B%BE%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C%E5%AF%BC%E8%AE%BA">图神经网络导论</h1>

<hr class="width50">
<h2 class="mume-header" id="%E7%9F%A9%E9%98%B5%E6%B1%82%E5%AF%BC">矩阵求导</h2>

<div class="bottom5"></div>
<h3 class="mume-header" id="%E8%AE%A1%E7%AE%97%E6%9C%BA%E7%A7%91%E5%AD%A6%E4%B8%8E%E6%8A%80%E6%9C%AF%E5%AD%A6%E9%99%A2-nbsp-nbsp-%E5%BC%A0%E8%85%BE">计算机科学与技术学院 &nbsp; &nbsp; 张腾</h3>

<br>
<h4 class="mume-header" id="tengzhanghusteducn"><a href="mailto:tengzhang@hust.edu.cn">tengzhang@hust.edu.cn</a></h4>

</section><section vertical="true" data-notes="" lineno="30" class="slide " data-line="30" data-h="0" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>机器学习中的矩阵求导</h5></div></div>
<p>监督学习中用梯度下降法求解 <strong>正则化项 + 损失函数</strong> 形式的问题</p>
<p>

$$
\begin{align*}
    \wv ~ \leftarrow ~ \wv - \eta ~ \class{blue}{\frac{\partial F(\wv)}{\partial \wv}}, \quad \Wv ~ \leftarrow ~ \Wv - \eta ~ \class{blue}{\frac{\partial F(\Wv)}{\partial \Wv}}
\end{align*}
$$
</p>

<p>如何求<span class="mathjax-exps">$\partial F(\wv) / \partial \wv$</span>和<span class="mathjax-exps">$\partial F(\Wv) / \partial \Wv$</span>？</p>
<div class="bottom4"></div>
<p>无监督学习中用极大似然法估计数据的分布，设<span class="mathjax-exps">$\Rbb^d \ni \xv \overset{\mathrm{IID}}{\sim} \Ncal (\xv | \muv, \Sigmav)$</span></p>
<p>

$$
\begin{align*}
    \max_{\muv, \Sigmav} \prod_{i \in [m]} \frac{1}{(2 \pi)^{d/2}} \frac{1}{|\Sigmav|^{1/2}} \exp \left( -\frac{1}{2} (\xv_i - \muv)^\top \Sigmav^{-1} (\xv_i - \muv) \right)
\end{align*}
$$
</p>

<p>如何求<span class="mathjax-exps">$\partial \muv^\top \Sigmav^{-1} \muv / \partial \muv$</span>、<span class="mathjax-exps">$\partial \ln |\Sigmav| / \partial \Sigmav$</span>、<span class="mathjax-exps">$\partial \muv^\top \Sigmav^{-1} \muv / \partial \Sigmav$</span>？</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="66" class="slide " data-line="66" data-h="1" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>9 种求导情形</h5></div></div>
<div class="fullborder column1-bold column2-bold column3-bold">
<table>
<thead>
<tr>
<th style="text-align:center"><span class="blue"><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>标量</span></th>
<th style="text-align:center"><span class="yellow"><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>向量</span></th>
<th style="text-align:center"><span class="yellow"><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>矩阵</span></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center"><span class="yellow"><span class="mathjax-exps">$\partial$</span>向量 /<span class="mathjax-exps">$\partial$</span>标量</span></td>
<td style="text-align:center"><span class="yellow"><span class="mathjax-exps">$\partial$</span>向量 /<span class="mathjax-exps">$\partial$</span>向量</span></td>
<td style="text-align:center"><span class="red"><span class="mathjax-exps">$\partial$</span>向量 /<span class="mathjax-exps">$\partial$</span>矩阵</span></td>
</tr>
<tr>
<td style="text-align:center"><span class="yellow"><span class="mathjax-exps">$\partial$</span>矩阵 /<span class="mathjax-exps">$\partial$</span>标量</span></td>
<td style="text-align:center"><span class="red"><span class="mathjax-exps">$\partial$</span>矩阵 /<span class="mathjax-exps">$\partial$</span>向量</span></td>
<td style="text-align:center"><span class="red"><span class="mathjax-exps">$\partial$</span>矩阵 /<span class="mathjax-exps">$\partial$</span>矩阵</span></td>
</tr>
</tbody>
</table>
</div>
<p><span class="blue"><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>标量</span>就是我们熟悉的单变量求导</p>
<p><span class="red"><span class="mathjax-exps">$\partial$</span>向量 /<span class="mathjax-exps">$\partial$</span>矩阵、<span class="mathjax-exps">$\partial$</span>矩阵 /<span class="mathjax-exps">$\partial$</span>向量、<span class="mathjax-exps">$\partial$</span>矩阵 /<span class="mathjax-exps">$\partial$</span>矩阵</span>会涉及高阶张量</p>
<p>我们考虑剩下的 5 种情形</p>
<ul>
<li><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>向量</li>
<li><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>矩阵</li>
<li><span class="mathjax-exps">$\partial$</span>向量 /<span class="mathjax-exps">$\partial$</span>标量</li>
<li><span class="mathjax-exps">$\partial$</span>矩阵 /<span class="mathjax-exps">$\partial$</span>标量</li>
<li><span class="mathjax-exps">$\partial$</span>向量 /<span class="mathjax-exps">$\partial$</span>向量</li>
</ul>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="93" class="slide " data-line="93" data-h="1" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵求导的分子布局</h5></div></div>
<p><span class="blue">标量对向量、矩阵的求导结果与分母转置尺寸相同</span></p>
<p>

$$
\begin{align*}
    \frac{\partial u}{\partial \xv} \triangleq \begin{bmatrix} \frac{\partial u}{\partial x_1} &amp; \ldots &amp; \frac{\partial u}{\partial x_l} \end{bmatrix}, \quad \frac{\partial u}{\partial \Xv} \triangleq \begin{bmatrix}
        \frac{\partial u}{\partial x_{11}} &amp; \ldots &amp; \frac{\partial u}{\partial x_{m1}} \\
        \vdots                                                      &amp; \ddots &amp; \vdots                             \\
        \frac{\partial u}{\partial x_{1n}} &amp; \ldots &amp; \frac{\partial u}{\partial x_{mn}}
    \end{bmatrix} \in \Rbb^{n \times m}
\end{align*}
$$
</p>

<div class="bottom4"></div>
<p><span class="blue">向量、矩阵对标量的求导结果与分子尺寸相同</span></p>
<p>

$$
\begin{align*}
    \frac{\partial \uv}{\partial x} \triangleq \begin{bmatrix}
        \frac{\partial u_1}{\partial x} \\ \vdots \\ \frac{\partial u_l}{\partial x}
    \end{bmatrix}, \quad
    \frac{\partial \Uv}{\partial x} \triangleq \begin{bmatrix}
        \frac{\partial u_{11}}{\partial x} &amp; \ldots &amp; \frac{\partial u_{1n}}{\partial x} \\
        \vdots                             &amp; \ddots &amp; \vdots                             \\
        \frac{\partial u_{m1}}{\partial x} &amp; \ldots &amp; \frac{\partial u_{mn}}{\partial x}
    \end{bmatrix} \in \Rbb^{m \times n}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="136" class="slide " data-line="136" data-h="1" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵求导的分子布局</h5></div></div>
<p>向量对向量求导的定义为<span class="blue">雅可比矩阵</span> (行看分子、列看分母)</p>
<p>

$$
\begin{align*}
    \frac{\partial \uv}{\partial \xv} \triangleq \begin{bmatrix}
        \frac{\partial u_1}{\partial x_1} &amp; \frac{\partial u_1}{\partial x_2} &amp; \ldots &amp; \frac{\partial u_1}{\partial x_m} \\
        \frac{\partial u_2}{\partial x_1} &amp; \frac{\partial u_2}{\partial x_2} &amp; \ldots &amp; \frac{\partial u_2}{\partial x_m} \\
        \vdots                            &amp; \vdots                            &amp; \ddots &amp; \vdots                            \\
        \frac{\partial u_l}{\partial x_1} &amp; \frac{\partial u_l}{\partial x_2} &amp; \ldots &amp; \frac{\partial u_l}{\partial x_m}
    \end{bmatrix}
    = \begin{bmatrix} \frac{\partial u_1}{\partial \xv} \\ \frac{\partial u_2}{\partial \xv} \\ \vdots \\ \frac{\partial u_l}{\partial \xv} \end{bmatrix}
    = \begin{bmatrix} \frac{\partial \uv}{\partial x_1} &amp; \frac{\partial \uv}{\partial x_2} &amp; \cdots &amp; \frac{\partial \uv}{\partial x_m} \end{bmatrix}
\end{align*}
$$
</p>

<div class="bottom2"></div>
<ul>
<li>分子布局的好处是<span class="blue">链式法则跟单变量求导中的顺序一样</span></li>
<li>分子布局的坏处是计算<span class="mathjax-exps">$\wv$</span>的梯度<span class="mathjax-exps">$\partial F(\wv) / \partial \wv$</span>时要<span class="blue">多做一个转置</span></li>
<li>分母布局的结果均是分子布局的转置，好处是算梯度时不用做转置，坏处是链式法则的顺序要完全反过来</li>
<li>两者结合为混合布局，算梯度时不做转置，但链式法则的顺序无固定规律，需根据矩阵尺寸仔细推敲，新手建议先学分子布局，熟练老鸟可直接混合布局</li>
</ul>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="168" class="slide " data-line="168" data-h="2" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>基本结果</h5></div></div>
<p>单变量求导中<span class="blue">常量的导数为零</span>：<span class="mathjax-exps">$\partial a / \partial x = 0$</span>，类似的这里有</p>
<p>

$$
\begin{align*}
    \frac{\partial \av}{\partial x} = \zerov, ~ \frac{\partial a}{\partial \xv} = \zerov^\top, ~ \frac{\partial \av}{\partial \xv} = \zerov, ~ \frac{\partial \Av}{\partial x} = \zerov, ~ \frac{\partial a}{\partial \Xv} = \zerov^\top
\end{align*}
$$
</p>

<p>单变量求导中<span class="blue">常数标量乘</span>的求导法则为<span class="mathjax-exps">$\frac{\partial (a u)}{\partial x} = a \frac{\partial u}{\partial x}$</span>，类似的这里有</p>
<p>

$$
\begin{align*}
    \frac{\partial a \uv}{\partial x} = a \frac{\partial \uv}{\partial x}, ~ \frac{\partial a u}{\partial \xv} = a \frac{\partial u}{\partial \xv}, ~ \frac{\partial a \uv}{\partial \xv} = a \frac{\partial \uv}{\partial \xv}, ~ \frac{\partial a \Uv}{\partial x} = a \frac{\partial \Uv}{\partial x}, ~ \frac{\partial a u}{\partial \Xv} = a \frac{\partial u}{\partial \Xv}
\end{align*}
$$
</p>

<p>单变量微积分中<span class="blue">加法</span>的求导法则为<span class="mathjax-exps">$\frac{\partial (u+v)}{\partial x} = \frac{\partial u}{\partial x} + \frac{\partial v}{\partial x}$</span>，类似的这里有</p>
<p>

$$
\begin{align*}
     &amp; \frac{\partial (\uv + \vv)}{\partial x} = \frac{\partial \uv}{\partial x} + \frac{\partial \vv}{\partial x}, ~ \frac{\partial (u+v)}{\partial \xv} = \frac{\partial u}{\partial \xv} + \frac{\partial v}{\partial \xv}, ~ \frac{\partial (\uv + \vv)}{\partial \xv} = \frac{\partial \uv}{\partial \xv} + \frac{\partial \vv}{\partial \xv} \\
     &amp; \frac{\partial (\Uv + \Vv)}{\partial x} = \frac{\partial \Uv}{\partial x} + \frac{\partial \Vv}{\partial x}, ~ \frac{\partial (u + v)}{\partial \Xv} = \frac{\partial u}{\partial \Xv} + \frac{\partial v}{\partial \Xv}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="211" class="slide " data-line="211" data-h="2" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>基本结果</h5></div></div>
<p>单变量求导中<span class="blue">乘法</span>的求导法则为<span class="mathjax-exps">$\frac{\partial (uv)}{\partial x} = \frac{\partial u}{\partial x} v + u \frac{\partial v}{\partial x}$</span>，类似的这里有</p>
<p>

$$
\begin{align*}
     &amp; \frac{\partial (\uv \vv)}{\partial x} = \frac{\partial \uv}{\partial x} \vv + \uv \frac{\partial \vv}{\partial x}, \quad \frac{\partial (uv)}{\partial \xv} = \frac{\partial u}{\partial \xv} v + u \frac{\partial v}{\partial \xv} \\
     &amp; \frac{\partial (\Uv \Vv)}{\partial x} = \frac{\partial \Uv}{\partial x} \Vv + \Uv \frac{\partial \Vv}{\partial x}, \quad \frac{\partial (uv)}{\partial \Xv} = \frac{\partial u}{\partial \Xv} v + u \frac{\partial v}{\partial \Xv}
\end{align*}
$$
</p>

<p>其中第一行可看作第二行的特例，第二行是因为</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial (\Uv \Vv)}{\partial x} \right]_{ij} &amp; = \sum_k \frac{\partial u_{ik}}{\partial x} v_{kj} + \sum_k u_{ik} \frac{\partial v_{kj}}{\partial x} = \left[ \frac{\partial \Uv}{\partial x} \Vv \right]_{ij} + \left[ \Uv \frac{\partial \Vv}{\partial x} \right]_{ij} \\
    \left[ \frac{\partial (uv)}{\partial \Xv} \right]_{ij}    &amp; = \frac{\partial (uv)}{\partial x_{ji}} = \frac{\partial u}{\partial x_{ji}} v + u \frac{\partial v}{\partial x_{ji}} = \left[ \frac{\partial u}{\partial \Xv} \right]_{ij} v + u \left[ \frac{\partial v}{\partial \Xv} \right]_{ij}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> <span class="mathjax-exps">$\partial (\uv^\top \vv) / \partial \xv$</span>属于<span class="yellow"><span class="mathjax-exps">$\partial$</span>标量 /<span class="mathjax-exps">$\partial$</span>向量</span>的情形，后面再讲</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="245" class="slide " data-line="245" data-h="2" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>基本结果</h5></div></div>
<p>单变量求导中有<span class="mathjax-exps">$\partial x / \partial x = 1$</span>，类似的这里有</p>
<p>

$$
\begin{align*}
    \frac{\partial x_i}{\partial \xv} = \ev_i^\top, \quad \frac{\partial \xv}{\partial x_i} = \ev_i, \quad \frac{\partial \xv}{\partial \xv} = \Iv, \quad \frac{\partial x_{ij}}{\partial \Xv} = \Ev_{ji}, \quad \frac{\partial \Xv}{\partial x_{ij}} = \Ev_{ij}
\end{align*}
$$
</p>

<p>其中</p>
<ul>
<li><span class="mathjax-exps">$\ev_i$</span>是第<span class="mathjax-exps">$i$</span>个元素为<span class="mathjax-exps">$1$</span>其余为<span class="mathjax-exps">$0$</span>的向量</li>
<li><span class="mathjax-exps">$\Ev_{ij}$</span>是<span class="mathjax-exps">$(i,j)$</span>处为<span class="mathjax-exps">$1$</span>其余为<span class="mathjax-exps">$0$</span>的矩阵</li>
</ul>
<div class="bottom4"></div>
<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 对神经网络第<span class="mathjax-exps">$l$</span>层<span class="mathjax-exps">$\zv_l = \Wv_l ~ \av_{l-1} + \bv_l$</span>，易知有</p>
<p>

$$
\begin{align*}
    \frac{\partial \zv_l}{\partial \bv_l} = \frac{\partial \bv_l}{\partial \bv_l} = \Iv
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="282" class="slide " data-line="282" data-h="3" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>链式法则</h5></div></div>
<p>单变量求导中的<span class="blue">链式法则</span>为<span class="mathjax-exps">$\frac{\partial g(u)}{\partial x} = \frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial x}$</span></p>
<p>情形一，只涉及向量：设<span class="mathjax-exps">$\xv \in \Rbb^n$</span>，<span class="mathjax-exps">$\uv = \uv(\xv) \in \Rbb^m$</span>，<span class="mathjax-exps">$\gv: \Rbb^m \mapsto \Rbb^l$</span>，则</p>
<p>

$$
\begin{align*}
    \underbrace{\class{blue}{\frac{\partial \gv(\uv)}{\partial \xv}}}_{l \times n} = \underbrace{\class{blue}{\frac{\partial \gv(\uv)}{\partial \uv}}}_{l \times m} \underbrace{\class{blue}{\frac{\partial \uv}{\partial \xv}}}_{m \times n}
\end{align*}
$$
</p>

<p>这是因为</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \gv(\uv)}{\partial \xv} \right]_{ij} &amp; = \frac{\partial [\gv(\uv)]_i}{\partial x_j} = \sum_{k \in [m]} \frac{\partial [\gv(\uv)]_i}{\partial u_k} \frac{\partial u_k}{\partial x_j} = \frac{\partial [\gv(\uv)]_i}{\partial \uv} \frac{\partial \uv}{\partial x_j} \\
    &amp; = \left[ \frac{\partial \gv(\uv)}{\partial \uv} \right]_{i,:} \left[ \frac{\partial \uv}{\partial \xv} \right]_{:,j} = \left[ \frac{\partial \gv(\uv)}{\partial \uv} \frac{\partial \uv}{\partial \xv} \right]_{ij}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 若<span class="mathjax-exps">$n = m = l = 1$</span>，就退化成了单变量的链式法则</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="317" class="slide " data-line="317" data-h="3" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>链式法则</h5></div></div>
<p>单变量求导中的<span class="blue">链式法则</span>为<span class="mathjax-exps">$\frac{\partial g(u)}{\partial x} = \frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial x}$</span></p>
<div class="bottom4"></div>
<p>情形二，自变量是矩阵：设<span class="mathjax-exps">$u = u(\Xv)$</span>，<span class="mathjax-exps">$g: \Rbb \mapsto \Rbb$</span>，则</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial g(u)}{\partial \Xv} \right]_{ij} = \frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial x_{ji}} = \frac{\partial g(u)}{\partial u} \left[ \frac{\partial u}{\partial \Xv} \right]_{ij} ~ \Longrightarrow ~ \class{blue}{\frac{\partial g(u)}{\partial \Xv} = \frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial \Xv}}
\end{align*}
$$
</p>

<div class="bottom4"></div>
<p>情形三，中间变量是矩阵：设<span class="mathjax-exps">$\Uv = \Uv(x) \in \Rbb^{m \times n}$</span>，<span class="mathjax-exps">$g: \Rbb^{m \times n} \mapsto \Rbb$</span>，则</p>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial g(\Uv)}{\partial x}} &amp; = \sum_{p,q} \frac{\partial g(\Uv)}{\partial u_{pq}} \frac{\partial u_{pq}}{\partial x} = \sum_{q,p} \left[ \frac{\partial g(\Uv)}{\partial \Uv} \right]_{qp} \left[ \frac{\partial \Uv}{\partial x} \right]_{pq} \\
    &amp; = \class{blue}{\tr \left( \frac{\partial g(\Uv)}{\partial \Uv} \frac{\partial \Uv}{\partial x} \right)}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="354" class="slide " data-line="354" data-h="4" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>大纲</h5></div></div>
<div class="sparse">
<p><span class="blue">向量对标量求导</span></p>
<p>标量对向量求导</p>
<p>向量对向量求导</p>
<p>矩阵对标量求导</p>
<p>标量对矩阵求导</p>
</div>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="374" class="slide " data-line="374" data-h="4" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>向量对标量求导</h5></div></div>
<p>矩阵和向量的乘积是向量，若<span class="mathjax-exps">$\Av$</span>与<span class="mathjax-exps">$\xv$</span>无关，易知有</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \Av \uv}{\partial x} \right]_{i} &amp; = \frac{\partial [\Av \uv]_i}{\partial x} = \frac{\partial \sum_k a_{ik} u_k}{\partial x} = \sum_k a_{ik} \frac{\partial u_k}{\partial x} = \left[ \Av \frac{\partial \uv}{\partial x} \right]_i \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial \Av \uv}{\partial x} = \Av \frac{\partial \uv}{\partial x}}
\end{align*}
$$
</p>

<p>于是</p>
<p>

$$
\begin{align*}
    \frac{\partial \uv^\top \Av}{\partial x} = \left( \frac{\partial \Av^\top \uv}{\partial x} \right)^\top = \left( \Av^\top \frac{\partial \uv}{\partial x} \right)^\top = \left( \frac{\partial \uv}{\partial x} \right)^\top \Av = \frac{\partial \uv^\top}{\partial x} \Av
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="405" class="slide " data-line="405" data-h="4" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>向量对标量求导</h5></div></div>
<p>向量的外积也是向量</p>
<p>记<span class="mathjax-exps">$\uv = [u_1(x); u_2(x); u_3(x)]$</span>，<span class="mathjax-exps">$\vv = [v_1(x); v_2(x); v_3(x)]$</span>，则</p>
<p>

$$
\begin{align*}
    \uv \times \vv = \begin{bmatrix}
        u_2 v_3 - u_3 v_2 \\ u_3 v_1 - u_1 v_3 \\ u_1 v_2 - u_2 v_1
    \end{bmatrix}
\end{align*}
$$
</p>

<p>于是</p>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial (\uv \times \vv)}{\partial x}} = \begin{bmatrix}
        \frac{\partial u_2}{\partial x} v_3 - \frac{\partial u_3}{\partial x} v_2 + u_2 \frac{\partial v_3}{\partial x} - u_3 \frac{\partial v_2}{\partial x} \\
        \frac{\partial u_3}{\partial x} v_1 - \frac{\partial u_1}{\partial x} v_3 + u_3 \frac{\partial v_1}{\partial x} - u_1 \frac{\partial v_3}{\partial x} \\
        \frac{\partial u_1}{\partial x} v_2 - \frac{\partial u_2}{\partial x} v_1 + u_1 \frac{\partial v_2}{\partial x} - u_2 \frac{\partial v_1}{\partial x} \\
    \end{bmatrix} = \class{blue}{\frac{\partial \uv}{\partial x} \times \vv + \uv \times \frac{\partial \vv}{\partial x}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="443" class="slide " data-line="443" data-h="5" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>大纲</h5></div></div>
<div class="sparse">
<p>向量对标量求导</p>
<p><span class="blue">标量对向量求导</span></p>
<p>向量对向量求导</p>
<p>矩阵对标量求导</p>
<p>标量对矩阵求导</p>
</div>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="463" class="slide " data-line="463" data-h="5" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>标量对向量求导</h5></div></div>
<p>二次型是标量，设<span class="mathjax-exps">$\Av$</span>与<span class="mathjax-exps">$\xv$</span>无关，易知有</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \uv^\top \Av \vv}{\partial \xv} \right]_i &amp; = \frac{\partial \sum_{j,k} u_j a_{jk} v_k}{\partial x_i} = \sum_{j,k} u_j a_{jk} \frac{\partial v_k}{\partial x_i} + \sum_{j,k} \frac{\partial u_j}{\partial x_i} a_{jk} v_k         \\
    &amp; = \uv^\top \Av \frac{\partial \vv}{\partial x_i} + \vv^\top \Av^\top \frac{\partial \uv}{\partial x_i} = \left[ \uv^\top \Av \frac{\partial \vv}{\partial \xv} \right]_i + \left[ \vv^\top \Av^\top \frac{\partial \uv}{\partial \xv} \right]_i \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial \uv^\top \Av \vv}{\partial \xv} = \uv^\top \Av \frac{\partial \vv}{\partial \xv} + \vv^\top \Av^\top \frac{\partial \uv}{\partial \xv}} \\
    &amp; \overset{\Av = \Iv}{\Longrightarrow} \frac{\partial \uv^\top \vv}{\partial \xv} = \uv^\top \frac{\partial \vv}{\partial \xv} + \vv^\top \frac{\partial \uv}{\partial \xv}
\end{align*}
$$
</p>

<p>进一步若<span class="mathjax-exps">$\uv = \av$</span>与<span class="mathjax-exps">$\xv$</span>无关，则</p>
<p>

$$
\begin{align*}
    \frac{\partial \av^\top \vv}{\partial \xv} = \av^\top \frac{\partial \vv}{\partial \xv}, \quad \frac{\partial \av^\top \xv}{\partial \xv} = \av^\top \frac{\partial \xv}{\partial \xv} = \av^\top, \quad \frac{\partial \bv^\top \Av \xv}{\partial \xv} = \bv^\top \Av
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="496" class="slide " data-line="496" data-h="5" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>标量对向量求导</h5></div></div>
<p>

$$
\begin{align*}
    \frac{\partial \uv^\top \Av \vv}{\partial \xv} &amp; = \uv^\top \Av \frac{\partial \vv}{\partial \xv} + \vv^\top \Av^\top \frac{\partial \uv}{\partial \xv} \\
    &amp; \overset{\uv = \vv = \xv}{~~ \Longrightarrow ~~} \frac{\partial \xv^\top \Av \xv}{\partial \xv} = \xv^\top \Av \frac{\partial \xv}{\partial \xv} + \xv^\top \Av^\top \frac{\partial \xv}{\partial \xv} = \xv^\top (\Av + \Av^\top) \\
    &amp; \overset{\Av = \bv \av^\top}{~~ \Longrightarrow ~~} \frac{\partial \xv^\top \bv \av^\top \xv}{\partial \xv} = \frac{\partial \av^\top \xv \xv^\top \bv}{\partial \xv} = \xv^\top (\av \bv^\top + \bv \av^\top) \\
    &amp; \overset{\Av = \Iv}{~~ \Longrightarrow ~~} \frac{\partial \xv^\top \xv}{\partial \xv} = \frac{\partial \|\xv\|_2^2}{\partial \xv} = 2 \xv^\top \\
\end{align*}
$$
</p>

<p>更一般的有</p>
<p>

$$
\begin{align*}
    \frac{\partial (\Av \xv + \bv)^\top \Cv (\Dv \xv + \ev)}{\partial \xv} &amp; = \frac{\partial (\xv^\top \Av^\top \Cv \Dv \xv + \bv^\top \Cv \Dv \xv + \xv^\top \Av^\top \Cv \ev + \bv^\top \ev)}{\partial \xv} \\
    &amp; = \xv^\top (\Av^\top \Cv \Dv + \Dv^\top \Cv^\top \Av) + \bv^\top \Cv \Dv + \ev^\top \Cv^\top \Av                                  \\
    &amp; = (\Dv \xv + \ev)^\top \Cv^\top \Av + (\Av \xv + \bv)^\top \Cv \Dv
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="529" class="slide " data-line="529" data-h="5" data-v="3">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>标量对向量求导</h5></div></div>
<p>范数也是标量，若<span class="mathjax-exps">$\av$</span>与<span class="mathjax-exps">$\xv$</span>无关，则</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \| \xv - \av \|_2}{\partial \xv} \right]_i &amp; = \frac{\partial \| \xv - \av \|_2}{\partial x_i} = \frac{\partial \sqrt{\sum_j (x_j - a_j)^2}}{\partial x_i} \\
    &amp; = \frac{1}{2} \frac{2 (x_i - a_i)}{\sqrt{\sum_j (x_j - a_j)^2}} = \frac{x_i - a_i}{\| \xv - \av \|_2} \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial \| \xv - \av \|_2}{\partial \xv} = \frac{(\xv - \av)^\top}{\| \xv - \av \|_2}}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 从这个例子可以看出为何<span class="mathjax-exps">$\ell_2$</span>正则项一般用平方的形式</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="551" class="slide " data-line="551" data-h="6" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>大纲</h5></div></div>
<div class="sparse">
<p>向量对标量求导</p>
<p>标量对向量求导</p>
<p><span class="blue">向量对向量求导</span></p>
<p>矩阵对标量求导</p>
<p>标量对矩阵求导</p>
</div>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="571" class="slide " data-line="571" data-h="6" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>向量对向量求导</h5></div></div>
<p>若<span class="mathjax-exps">$\Av$</span>与<span class="mathjax-exps">$\xv$</span>无关，前面已得向量对标量的求导结果<span class="mathjax-exps">$\frac{\partial \Av \uv}{\partial x} = \Av \frac{\partial \uv}{\partial x}$</span>，于是</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \Av \uv}{\partial \xv} \right]_{:,j} = \frac{\partial \Av \uv}{\partial x_j} = \Av \frac{\partial \uv}{\partial x_j} = \left[ \Av \frac{\partial \uv}{\partial \xv} \right]_{:,j} &amp; \Longrightarrow \class{blue}{\frac{\partial \Av \uv}{\partial \xv} = \Av \frac{\partial \uv}{\partial \xv}} \\
    &amp; \overset{\uv = \xv}{\Longrightarrow} \frac{\partial \Av \xv}{\partial \xv} = \Av \frac{\partial \xv}{\partial \xv} = \Av
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 对神经网络第<span class="mathjax-exps">$l$</span>层<span class="mathjax-exps">$\zv_l = \Wv_l ~ \av_{l-1} + \bv_l$</span>，易知有<span class="mathjax-exps">$\frac{\partial \zv_l}{\partial \av_{l-1}} = \frac{\partial (\Wv_l ~ \av_{l-1})}{\partial \av_{l-1}} = \Wv_l$</span></p>
<div class="bottom2"></div>
<p>若<span class="mathjax-exps">$v = v(\xv)$</span>，则</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial v \uv}{\partial \xv} \right]_{ij} &amp; = \frac{\partial v u_i}{\partial x_j} = v \frac{\partial u_i}{\partial x_j} + u_i \frac{\partial v}{\partial x_j} = v \left[ \frac{\partial \uv}{\partial \xv} \right]_{ij} + \left[ \uv \frac{\partial v}{\partial \xv} \right]_{ij} \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial v \uv}{\partial \xv} = v \frac{\partial \uv}{\partial \xv} + \uv \frac{\partial v}{\partial \xv}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="607" class="slide " data-line="607" data-h="6" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>向量对向量求导</h5></div></div>
<p>若<span class="mathjax-exps">$\av$</span>与<span class="mathjax-exps">$\xv$</span>无关，结合</p>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial \| \xv - \av \|_2}{\partial \xv^\top} = \frac{\xv - \av}{\| \xv - \av \|_2}}
\end{align*}
$$
</p>

<p>可得<span class="mathjax-exps">$\ell_2$</span>范数的二阶导 (海森矩阵)</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial}{\partial \xv} \frac{\xv - \av}{\| \xv - \av \|_2} \right]_{ij} = \frac{\partial}{\partial x_j} \frac{x_i - a_i}{\| \xv - \av \|_2} \\
    &amp; \qquad = \frac{\delta_{ij} \|\xv - \av\|_2}{\| \xv - \av \|_2^2} - \frac{x_i - a_i}{\| \xv - \av \|_2^2} \frac{\partial \| \xv - \av \|_2}{\partial x_j} \\
    &amp; \qquad = \frac{\delta_{ij}}{\| \xv - \av \|_2} - \frac{x_i - a_i}{\| \xv - \av \|_2^2} \frac{x_j - a_j}{\| \xv - \av \|_2} \\
    &amp; \qquad \Longrightarrow \class{blue}{\frac{\partial^2 \| \xv - \av \|_2}{\partial \xv \partial \xv^\top} = \frac{\partial}{\partial \xv} \frac{\xv - \av}{\| \xv - \av \|_2} = \frac{\Iv}{\| \xv - \av \|_2} - \frac{(\xv - \av)(\xv - \av)^\top}{\| \xv - \av \|_2^3}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="640" class="slide " data-line="640" data-h="7" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>大纲</h5></div></div>
<div class="sparse">
<p>向量对标量求导</p>
<p>标量对向量求导</p>
<p>向量对向量求导</p>
<p><span class="blue">矩阵对标量求导</span></p>
<p>标量对矩阵求导</p>
</div>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="660" class="slide " data-line="660" data-h="7" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>若<span class="mathjax-exps">$u = u(x)$</span>，<span class="mathjax-exps">$\Vv = \Vv(x)$</span>，则</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial u \Vv}{\partial x} \right]_{ij} &amp; = \frac{\partial u v_{ij}}{\partial x} = \frac{\partial u}{\partial x} v_{ij} + u \frac{\partial v_{ij}}{\partial x} = \frac{\partial u}{\partial x} \left[ \Vv \right]_{ij} + u \left[ \frac{\partial \Vv}{\partial x} \right]_{ij} \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial u \Vv}{\partial x} = \frac{\partial u}{\partial x} \Vv + u \frac{\partial \Vv}{\partial x}}
\end{align*}
$$
</p>

<p>若乘积求导法则中的<span class="mathjax-exps">$\Uv$</span>或<span class="mathjax-exps">$\Vv$</span>可继续分解，则</p>
<p>

$$
\begin{align*}
    \frac{\partial (\Uv \Vv)}{\partial x} &amp; = \frac{\partial \Uv}{\partial x} \Vv + \Uv \frac{\partial \Vv}{\partial x} \\
    &amp; \Downarrow \\
    \class{blue}{\frac{\partial (\Uv \Vv \Wv)}{\partial x}} &amp; = \frac{\partial \Uv}{\partial x} \Vv \Wv + \Uv \frac{\partial \Vv \Wv}{\partial x} = \frac{\partial \Uv}{\partial x} \Vv \Wv + \Uv \left( \frac{\partial \Vv}{\partial x} \Wv + \Vv \frac{\partial \Wv}{\partial x} \right) \\
    &amp; = \class{blue}{\frac{\partial \Uv}{\partial x} \Vv \Wv + \Uv \frac{\partial \Vv}{\partial x} \Wv + \Uv \Vv \frac{\partial \Wv}{\partial x}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="694" class="slide " data-line="694" data-h="7" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>

$$
\begin{align}
    \class{blue}{\frac{\partial (\Uv \Vv \Wv)}{\partial x} = \frac{\partial \Uv}{\partial x} \Vv \Wv + \Uv \frac{\partial \Vv}{\partial x} \Wv + \Uv \Vv \frac{\partial \Wv}{\partial x}}
\end{align}
$$
</p>

<p>由此可知若<span class="mathjax-exps">$\Av$</span>、<span class="mathjax-exps">$\Bv$</span>与<span class="mathjax-exps">$x$</span>无关，则</p>
<p>

$$
\begin{align*}
    \frac{\partial \Av \Uv \Bv}{\partial x} = \Av \frac{\partial \Uv}{\partial x} \Bv
\end{align*}
$$
</p>

<p>当<span class="mathjax-exps">$\Uv$</span>为方阵、<span class="mathjax-exps">$n$</span>为正整数时有</p>
<p>

$$
\begin{align}
    \class{blue}{\frac{\partial \Uv^n}{\partial x}} &amp; = \Uv^{n-1} \frac{\partial \Uv}{\partial x} + \Uv^{n-2} \frac{\partial \Uv}{\partial x} \Uv + \cdots + \Uv \frac{\partial \Uv}{\partial x} \Uv^{n-2} + \frac{\partial \Uv}{\partial x} \Uv^{n-1} \\
    &amp; = \class{blue}{\sum_{i \in [n]} \Uv^{i-1} \frac{\partial \Uv}{\partial x} \Uv^{n-i}}
\end{align}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="735" class="slide " data-line="735" data-h="8" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>令乘积求导法则中的<span class="mathjax-exps">$\Vv = \Uv^{-1}$</span>可得</p>
<p>

$$
\begin{align} \label{eq: inverse}
    \zerov = \frac{\partial \Iv}{\partial x} &amp; = \frac{\partial \Uv \Uv^{-1}}{\partial x} = \Uv \frac{\partial \Uv^{-1}}{\partial x} + \frac{\partial \Uv}{\partial x} \Uv^{-1} \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial \Uv^{-1}}{\partial x} = - \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1}}
\end{align}
$$
</p>

<p>结合链式法则情形三<span class="mathjax-exps">$\frac{\partial g(\Uv)}{\partial x} = \tr ( \frac{\partial g(\Uv)}{\partial \Uv} \frac{\partial \Uv}{\partial x} )$</span>可知</p>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial [\Xv^{-1}]_{kl}}{\partial x_{ij}}} &amp; = \tr \left( \frac{\partial [\Xv^{-1}]_{kl}}{\partial \Xv^{-1}} \frac{\partial \Xv^{-1}}{\partial x_{ij}} \right) = - \tr \left( \Ev_{lk} \Xv^{-1} \frac{\partial \Xv}{\partial x_{ij}} \Xv^{-1} \right) \\
    &amp; = - \tr ( \Xv^{-1} \Ev_{lk} \Xv^{-1} \Ev_{ij} ) = - [\Xv^{-1} \Ev_{lk} \Xv^{-1}]_{ji} \\
    &amp; = - \sum_{p,q} [\Xv^{-1}]_{jp} [\Ev_{lk}]_{pq} [\Xv^{-1}]_{qi} = \class{blue}{- [\Xv^{-1}]_{jl} [\Xv^{-1}]_{ki}}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 遍历<span class="mathjax-exps">$\Xv^{-1}$</span>的下标<span class="mathjax-exps">$kl$</span>即可得到<span class="mathjax-exps">$\Xv^{-1}$</span>对<span class="mathjax-exps">$\Xv$</span>每个元素<span class="mathjax-exps">$x_{ij}$</span>的导数</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="770" class="slide " data-line="770" data-h="8" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial \Uv^{-1}}{\partial x} = - \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1}}
\end{align*}
$$
</p>

<p>结合<span class="mathjax-exps">$\frac{\partial (\Uv \Vv \Wv)}{\partial x} = \frac{\partial \Uv}{\partial x} \Vv \Wv + \Uv \frac{\partial \Vv}{\partial x} \Wv + \Uv \Vv \frac{\partial \Wv}{\partial x}$</span>可得海森矩阵</p>
<p>

$$
\begin{align*}
    &amp; \class{blue}{\frac{\partial^2 \Uv^{-1}}{\partial x \partial y}} = \frac{\partial}{\partial y} \left( - \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1} \right) \\
    &amp; \qquad = - \frac{\partial \Uv^{-1}}{\partial y} \frac{\partial \Uv}{\partial x} \Uv^{-1} - \Uv^{-1} \frac{\partial^2 \Uv}{\partial x \partial y} \Uv^{-1} - \Uv^{-1} \frac{\partial \Uv}{\partial x} \frac{\partial \Uv^{-1}}{\partial y} \\
    &amp; \qquad = \Uv^{-1} \frac{\partial \Uv}{\partial y} \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1} - \Uv^{-1} \frac{\partial^2 \Uv}{\partial x \partial y} \Uv^{-1} + \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1} \frac{\partial \Uv}{\partial y} \Uv^{-1}                                                                          \\
    &amp; \qquad = \class{blue}{\Uv^{-1} \left( \frac{\partial \Uv}{\partial y} \Uv^{-1} \frac{\partial \Uv}{\partial x} - \frac{\partial^2 \Uv}{\partial x \partial y} + \frac{\partial \Uv}{\partial x} \Uv^{-1} \frac{\partial \Uv}{\partial y} \right) \Uv^{-1}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="801" class="slide " data-line="801" data-h="9" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>矩阵除了常规的乘积外，还有哈达玛积和克罗内克积</p>
<br>
<p>设<span class="mathjax-exps">$\Uv, \Vv \in \Rbb^{m \times n}$</span>，则</p>
<p>

$$
\begin{align*}
    &amp; \class{blue}{\frac{\partial (\Uv \circ \Vv)}{\partial x}} = \begin{bmatrix}
        \frac{\partial u_{11} v_{11}}{\partial x} &amp; \cdots &amp; \frac{\partial u_{1n} v_{1n}}{\partial x} \\
        \vdots                                    &amp; \ddots &amp; \vdots                                    \\
        \frac{\partial u_{m1} v_{m1}}{\partial x} &amp; \cdots &amp; \frac{\partial u_{mn} v_{mn}}{\partial x} \\
    \end{bmatrix}                                                                   \\
    &amp; \qquad = \begin{bmatrix}
        \frac{\partial u_{11}}{\partial x} v_{11} &amp; \cdots &amp; \frac{\partial u_{1n}}{\partial x} v_{1n} \\
        \vdots                                    &amp; \ddots &amp; \vdots                                    \\
        \frac{\partial u_{m1}}{\partial x} v_{m1} &amp; \cdots &amp; \frac{\partial u_{mn}}{\partial x} v_{mn} \\
    \end{bmatrix} + \begin{bmatrix}
        u_{11} \frac{\partial v_{11}}{\partial x} &amp; \cdots &amp; u_{1n} \frac{\partial v_{1n}}{\partial x} \\
        \vdots                                    &amp; \ddots &amp; \vdots                                    \\
        u_{m1} \frac{\partial v_{m1}}{\partial x} &amp; \cdots &amp; u_{mn} \frac{\partial v_{mn}}{\partial x} \\
    \end{bmatrix}                                      \\
    &amp; \qquad = \class{blue}{\frac{\partial \Uv}{\partial x} \circ \Vv + \Uv \circ \frac{\partial \Vv}{\partial x}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="837" class="slide " data-line="837" data-h="9" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>

$$
\begin{align*}
    &amp; \class{blue}{\frac{\partial (\Uv \otimes \Vv)}{\partial x}} = \begin{bmatrix}
        \frac{\partial u_{11} \Vv}{\partial x} &amp; \cdots &amp; \frac{\partial u_{1n} \Vv}{\partial x} \\
        \vdots                                 &amp; \ddots &amp; \vdots                                 \\
        \frac{\partial u_{m1} \Vv}{\partial x} &amp; \cdots &amp; \frac{\partial u_{mn} \Vv}{\partial x} \\
    \end{bmatrix}                                                                       \\
    &amp; \qquad = \begin{bmatrix}
        \frac{\partial u_{11}}{\partial x} \Vv + u_{11} \frac{\partial \Vv}{\partial x} &amp; \cdots &amp; \frac{\partial u_{1n}}{\partial x} \Vv + u_{1n} \frac{\partial \Vv}{\partial x} \\
        \vdots                                                                          &amp; \ddots &amp; \vdots                                                                          \\
        \frac{\partial u_{m1}}{\partial x} \Vv + u_{m1} \frac{\partial \Vv}{\partial x} &amp; \cdots &amp; \frac{\partial u_{mn}}{\partial x} \Vv + u_{mn} \frac{\partial \Vv}{\partial x} \\
    \end{bmatrix}                                                                       \\
    &amp; \qquad = \begin{bmatrix}
        \frac{\partial u_{11}}{\partial x} \Vv &amp; \cdots &amp; \frac{\partial u_{1n}}{\partial x} \Vv \\
        \vdots                                 &amp; \ddots &amp; \vdots                                 \\
        \frac{\partial u_{m1}}{\partial x} \Vv &amp; \cdots &amp; \frac{\partial u_{mn}}{\partial x} \Vv \\
    \end{bmatrix} + \begin{bmatrix}
        u_{11} \frac{\partial \Vv}{\partial x} &amp; \cdots &amp; u_{1n} \frac{\partial \Vv}{\partial x} \\
        \vdots                                 &amp; \ddots &amp; \vdots                                 \\
        u_{m1} \frac{\partial \Vv}{\partial x} &amp; \cdots &amp; u_{mn} \frac{\partial \Vv}{\partial x} \\
    \end{bmatrix}                                          \\
    &amp; \qquad = \class{blue}{\frac{\partial \Uv}{\partial x} \otimes \Vv + \Uv \otimes \frac{\partial \Vv}{\partial x}}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="872" class="slide " data-line="872" data-h="9" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>矩阵对标量求导</h5></div></div>
<p>设<span class="mathjax-exps">$g(x) = a_0 + a_1 x + a_2 x^2 + \cdots$</span>，若<span class="mathjax-exps">$\Av$</span>为与<span class="mathjax-exps">$x$</span>无关的方阵，记</p>
<p>

$$
\begin{align*}
    g (x \Av)  &amp; = a_0 \Iv + a_1 x \Av + a_2 x^2 \Av^2 + a_3 x^3 \Av^3 + \cdots \\
    g' (x \Av) &amp; = a_1 \Iv + 2 a_2 x \Av + 3 a_3 x^2 \Av^2 + \cdots
\end{align*}
$$
</p>

<p>易知有</p>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial g(x \Av)}{\partial x}} &amp; = a_1 \Av + 2 a_2 x \Av^2 + 3 a_3 x^2 \Av^3 + \cdots                             \\
    &amp; = \Av (a_1 \Iv + 2 a_2 x \Av + 3 a_3 x^2 \Av^2 + \cdots) = \class{blue}{\Av g' (x \Av)} \\
    &amp; = (a_1 \Iv + 2 a_2 x \Av + 3 a_3 x^2 \Av^2 + \cdots) \Av = \class{blue}{g' (x \Av) \Av}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 对于<span class="mathjax-exps">$e^x$</span>、<span class="mathjax-exps">$\sin x$</span>、<span class="mathjax-exps">$\cos x$</span>，上式依然适用，例如</p>
<p>

$$
\begin{align*}
    \frac{\partial e^{x \Av}}{\partial x} = \Av e^{x \Av} = e^{x \Av} \Av, \quad \frac{\partial \sin (x \Av)}{\partial x} = \Av \cos (x \Av) = \cos (x \Av) \Av
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="917" class="slide " data-line="917" data-h="10" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>大纲</h5></div></div>
<div class="sparse">
<p>向量对标量求导</p>
<p>标量对向量求导</p>
<p>向量对向量求导</p>
<p>矩阵对标量求导</p>
<p><span class="blue">标量对矩阵求导</span></p>
</div>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="937" class="slide " data-line="937" data-h="10" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>矩阵常见的标量函数有<span class="blue">迹</span>和<span class="blue">行列式</span>，二次型可以归为迹来处理</p>
<br>
<p>若<span class="mathjax-exps">$a$</span>与<span class="mathjax-exps">$\Xv$</span>无关，<span class="mathjax-exps">$\Uv = \Uv(\Xv)$</span>，<span class="mathjax-exps">$\Vv = \Vv(\Xv)$</span>，则以下结论是显然的：</p>
<p>

$$
\begin{align*}
    \frac{\partial \tr(\Xv)}{\partial \Xv} = \Iv, ~ \frac{\partial \tr(\Uv+\Vv)}{\partial \Xv} = \frac{\partial \tr(\Uv)}{\partial \Xv} + \frac{\partial \tr(\Vv)}{\partial \Xv}, ~ \frac{\partial \tr(a \Uv)}{\partial \Xv} = a \frac{\partial \tr(\Uv)}{\partial \Xv}
\end{align*}
$$
</p>

<p>对于乘积有</p>
<p>

$$
\begin{align*}
    \class{blue}{\left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij}} &amp; = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \sum_{p,q} \left( \frac{\partial u_{pq}}{\partial x_{ji}} v_{qp} + u_{pq} \frac{\partial v_{qp}}{\partial x_{ji}} \right) \\
    &amp; = \tr \left( \frac{\partial \Uv}{\partial x_{ji}} \Vv \right) + \tr \left( \Uv \frac{\partial \Vv}{\partial x_{ji}} \right) = \class{blue}{\tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 由此可知<span class="blue">迹和求导的顺序可以交换</span></p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="974" class="slide " data-line="974" data-h="10" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Bv \Av$</span>与<span class="mathjax-exps">$\Xv$</span>无关，<span class="mathjax-exps">$\Vv = \Xv$</span>，则</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Bv \Av \Xv)}{\partial \Xv} \right]_{ij} &amp; = \tr \left( \Bv \Av \frac{\partial \Xv}{\partial x_{ji}} \right) = \tr ( \Bv \Av \Ev_{ji} ) = [\Bv \Av]_{ij} \\
    &amp; \Longrightarrow \frac{\partial \tr(\Bv \Av \Xv)}{\partial \Xv} = \frac{\partial \tr(\Av \Xv \Bv)}{\partial \Xv} = \frac{\partial \tr(\Xv \Bv \Av)}{\partial \Xv} = \Bv \Av
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Bv \Av$</span>与<span class="mathjax-exps">$\Xv$</span>无关，<span class="mathjax-exps">$\Vv = \Xv^\top$</span>，则</p>
<p>

$$
\begin{align*}
    \frac{\partial \tr(\Av \Xv^\top \Bv)}{\partial \Xv} = \frac{\partial \tr(\Xv^\top \Bv \Av)}{\partial \Xv} = \frac{\partial \tr(\Bv \Av \Xv^\top)}{\partial \Xv} = \frac{\partial \tr(\Xv \Av^\top \Bv^\top)}{\partial \Xv} = \Av^\top \Bv^\top
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="1015" class="slide " data-line="1015" data-h="11" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Av$</span>与<span class="mathjax-exps">$\Xv$</span>无关，<span class="mathjax-exps">$\Vv = \Xv \Xv^\top$</span>，则</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Av \Xv \Xv^\top)}{\partial \Xv} \right]_{ij} &amp; = \tr \left( \Av \frac{\partial \Xv \Xv^\top}{\partial x_{ji}} \right) = \tr \left( \Av \frac{\partial \Xv}{\partial x_{ji}} \Xv^\top \right) + \tr \left( \Av \Xv \frac{\partial \Xv^\top}{\partial x_{ji}} \right) \\
    &amp; = \tr(\Av \Ev_{ji} \Xv^\top) + \tr(\Av \Xv \Ev_{ij}) = [\Xv^\top \Av]_{ij} + [\Av \Xv]_{ji}
\end{align*}
$$
</p>

<p>从而</p>
<p>

$$
\begin{align*}
    &amp; \frac{\partial \tr(\Av \Xv \Xv^\top)}{\partial \Xv} = \frac{\partial \tr(\Xv^\top \Av \Xv)}{\partial \Xv} = \frac{\partial \tr(\Xv \Xv^\top \Av)}{\partial \Xv} = \Xv^\top (\Av + \Av^\top) \\
    \overset{\Xv \leftarrow \Xv^\top}{\Longrightarrow} &amp; \frac{\partial \tr(\Av \Xv^\top \Xv)}{\partial \Xv} = \frac{\partial \tr(\Xv \Av \Xv^\top)}{\partial \Xv} = \frac{\partial \tr(\Xv^\top \Xv \Av)}{\partial \Xv} = (\Av + \Av^\top) \Xv^\top
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1057" class="slide " data-line="1057" data-h="11" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Bv \Av$</span>与<span class="mathjax-exps">$\Xv$</span>无关，<span class="mathjax-exps">$\Vv = \Xv^{-1}$</span>，结合</p>
<p>

$$
\begin{align*}
    \frac{\partial \Uv^{-1}}{\partial x} = - \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1}
\end{align*}
$$
</p>

<p>可得</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial \tr(\Bv \Av \Xv^{-1})}{\partial \Xv} \right]_{ij} = \tr \left( \Bv \Av \frac{\partial \Xv^{-1}}{\partial x_{ji}} \right) = \tr \left( - \Bv \Av \Xv^{-1} \frac{\partial \Xv}{\partial x_{ji}} \Xv^{-1} \right) \\
    &amp; \quad = - \tr \left( \Xv^{-1} \Bv \Av \Xv^{-1} \Ev_{ji} \right) = - [\Xv^{-1} \Bv \Av \Xv^{-1}]_{ij} \\
    &amp; \quad \Longrightarrow \frac{\partial \tr(\Bv \Av \Xv^{-1})}{\partial \Xv} = \frac{\partial \tr(\Av \Xv^{-1} \Bv)}{\partial \Xv} = \frac{\partial \tr(\Xv^{-1} \Bv \Av)}{\partial \Xv} = - \Xv^{-1} \Bv \Av \Xv^{-1}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="1099" class="slide " data-line="1099" data-h="12" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Iv$</span>，<span class="mathjax-exps">$\Vv = (\Xv + \Av)^{-1}$</span>，结合<span class="mathjax-exps">$\frac{\partial \Uv^{-1}}{\partial x} = - \Uv^{-1} \frac{\partial \Uv}{\partial x} \Uv^{-1}$</span>可得</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Xv + \Av)^{-1}}{\partial \Xv} \right]_{ij} &amp; = \tr \left( \frac{\partial (\Xv + \Av)^{-1}}{\partial x_{ji}} \right) \\
    &amp; = - \tr \left( (\Xv + \Av)^{-1} \frac{\partial (\Xv + \Av)}{\partial x_{ji}} (\Xv + \Av)^{-1} \right) \\
    &amp; = - \tr \left( (\Xv + \Av)^{-1} (\Xv + \Av)^{-1} \Ev_{ji}  \right) \\
    &amp; = - [(\Xv + \Av)^{-1} (\Xv + \Av)^{-1}]_{ij} \\
    &amp; \Longrightarrow \frac{\partial \tr(\Xv + \Av)^{-1}}{\partial \Xv} = - (\Xv + \Av)^{-1} (\Xv + \Av)^{-1}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1131" class="slide " data-line="1131" data-h="12" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Av \Xv \Bv$</span>，<span class="mathjax-exps">$\Vv = \Xv^\top \Cv$</span>，其中<span class="mathjax-exps">$\Av$</span>、<span class="mathjax-exps">$\Bv$</span>、<span class="mathjax-exps">$\Cv$</span>与<span class="mathjax-exps">$\Xv$</span>无关，则</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial \tr(\Av \Xv \Bv \Xv^\top \Cv)}{\partial \Xv} \right]_{ij} = \tr \left( \frac{\partial \Av \Xv \Bv}{\partial x_{ji}} \Xv^\top \Cv \right) + \tr \left( \Av \Xv \Bv \frac{\partial \Xv^\top \Cv}{\partial x_{ji}} \right) \\
    &amp; \qquad = \tr \left( \Av \Ev_{ji} \Bv \Xv^\top \Cv \right) + \tr \left( \Av \Xv \Bv \Ev_{ij} \Cv \right) = [\Bv \Xv^\top \Cv \Av]_{ij} + [\Cv \Av \Xv \Bv]_{ji} \\
    &amp; \qquad \Longrightarrow \frac{\partial \tr(\Av \Xv \Bv \Xv^\top \Cv)}{\partial \Xv} = \Bv \Xv^\top \Cv \Av + \Bv^\top \Xv^\top \Av^\top \Cv^\top
\end{align*}
$$
</p>

<p>令<span class="mathjax-exps">$\Xv \leftarrow \Xv^\top$</span>，于是只需对结果取转置即有</p>
<p>

$$
\begin{align*}
    \frac{\partial \tr(\Av \Xv^\top \Bv \Xv \Cv)}{\partial \Xv} = \Cv \Av \Xv^\top \Bv + \Av^\top \Cv^\top \Xv^\top \Bv^\top
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1173" class="slide " data-line="1173" data-h="12" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Av \Xv \Bv$</span>，<span class="mathjax-exps">$\Vv = \Xv^\top \Cv$</span>，其中<span class="mathjax-exps">$\Av$</span>、<span class="mathjax-exps">$\Bv$</span>、<span class="mathjax-exps">$\Cv$</span>与<span class="mathjax-exps">$\Xv$</span>无关，则</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial \tr(\Av \Xv \Bv \Xv^\top \Cv)}{\partial \Xv} \right]_{ij} = \tr \left( \frac{\partial \Av \Xv \Bv}{\partial x_{ji}} \Xv^\top \Cv \right) + \tr \left( \Av \Xv \Bv \frac{\partial \Xv^\top \Cv}{\partial x_{ji}} \right) \\
    &amp; \qquad = \tr \left( \Av \Ev_{ji} \Bv \Xv^\top \Cv \right) + \tr \left( \Av \Xv \Bv \Ev_{ij} \Cv \right) = [\Bv \Xv^\top \Cv \Av]_{ij} + [\Cv \Av \Xv \Bv]_{ji} \\
    &amp; \qquad \Longrightarrow \frac{\partial \tr(\Av \Xv \Bv \Xv^\top \Cv)}{\partial \Xv} = \Bv \Xv^\top \Cv \Av + \Bv^\top \Xv^\top \Av^\top \Cv^\top
\end{align*}
$$
</p>

<p>令<span class="mathjax-exps">$\Xv \leftarrow \Xv^\top$</span>，于是只需对结果取转置即有</p>
<p>

$$
\begin{align*}
    \frac{\partial \tr(\Av \Xv^\top \Bv \Xv \Cv)}{\partial \Xv} = \Cv \Av \Xv^\top \Bv + \Av^\top \Cv^\top \Xv^\top \Bv^\top
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="1215" class="slide " data-line="1215" data-h="13" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Bv \Av$</span>与<span class="mathjax-exps">$\Xv$</span>无关，<span class="mathjax-exps">$\Vv = \Xv^n$</span>，其中<span class="mathjax-exps">$n$</span>是正整数，于是</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial \tr(\Bv \Av \Xv^n)}{\partial \Xv} \right]_{ij} = \tr \left( \Bv \Av \frac{\partial \Xv^n}{\partial x_{ji}} \right) = \tr \left( \Bv \Av \sum_{k \in [n]} \Xv^{k-1} \frac{\partial \Xv}{\partial x_{ji}} \Xv^{n-k} \right) \\
    &amp; ~ = \sum_{k \in [n]} \tr \left( \Bv \Av \Xv^{k-1} \frac{\partial \Xv}{\partial x_{ji}} \Xv^{n-k} \right) \\
    &amp; ~ = \sum_{k \in [n]} \tr ( \Xv^{n-k} \Bv \Av \Xv^{k-1} \Ev_{ji} ) = \sum_{k \in [n]} [\Xv^{n-k} \Bv \Av \Xv^{k-1}]_{ij} \\
    &amp; ~ \Longrightarrow \frac{\partial \tr(\Bv \Av \Xv^n)}{\partial \Xv} = \frac{\partial \tr(\Av \Xv^n \Bv)}{\partial \Xv} = \frac{\partial \tr(\Xv^n \Bv \Av)}{\partial \Xv} = \sum_{k \in [n]} \Xv^{n-k} \Bv \Av \Xv^{k-1}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1246" class="slide " data-line="1246" data-h="13" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \frac{\partial \tr(\Bv \Av \Xv^n)}{\partial \Xv} = \frac{\partial \tr(\Av \Xv^n \Bv)}{\partial \Xv} = \frac{\partial \tr(\Xv^n \Bv \Av)}{\partial \Xv} = \sum_{k \in [n]} \Xv^{n-k} \Bv \Av \Xv^{k-1}
\end{align*}
$$
</p>

<p>进一步若<span class="mathjax-exps">$\Av = \Bv = \Iv$</span>，则</p>
<p>

$$
\begin{align*}
    \frac{\partial \tr(\Xv^n)}{\partial \Xv} = \sum_{k \in [n]} \Xv^{n-k} \Xv^{k-1} = \sum_{k \in [n]} \Xv^{n-1} = n \Xv^{n-1}
\end{align*}
$$
</p>

<p><span class="yellow"><i class="fa fa-weixin" aria-hidden="true"></i></span> 形式上和单变量求导公式是一样的，类似的记</p>
<p>

$$
\begin{align*}
    e^{\Xv}  &amp; = \Iv + \Xv + \frac{\Xv^2}{2!} + \frac{\Xv^3}{3!} + \cdots              \\
    \sin \Xv &amp; = \Xv - \frac{\Xv^3}{3!} + \frac{\Xv^5}{5!} - \cdots, \quad \cos \Xv = \Iv - \frac{\Xv^2}{2!} + \frac{\Xv^4}{4!} - \frac{\Xv^6}{6!} + \cdots
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1287" class="slide " data-line="1287" data-h="13" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \frac{\partial \tr(e^{\Xv})}{\partial \Xv} &amp; = \frac{\partial }{\partial \Xv} \tr \left( \Iv + \Xv + \frac{\Xv^2}{2!} + \frac{\Xv^3}{3!} + \cdots \right) \\
    &amp; = \frac{\partial \tr (\Iv)}{\partial \Xv} + \frac{\partial \tr (\Xv)}{\partial \Xv} + \frac{1}{2!} \frac{\partial \tr (\Xv^2)}{\partial \Xv} + \frac{1}{3!} \frac{\partial \tr (\Xv^3)}{\partial \Xv} + \cdots \\
    &amp; = \Iv + \Xv + \frac{\Xv^2}{2!} + \cdots = e^{\Xv} \\
    \frac{\partial \tr(\sin \Xv)}{\partial \Xv} &amp; = \frac{\partial }{\partial \Xv} \tr \left( \Xv - \frac{\Xv^3}{3!} + \frac{\Xv^5}{5!} - \cdots \right) \\
    &amp; = \Iv - \frac{\Xv^2}{2!} + \frac{\Xv^4}{4!} - \cdots = \cos \Xv \\
    \frac{\partial \tr(\cos \Xv)}{\partial \Xv} &amp; = \frac{\partial }{\partial \Xv} \tr \left( \Iv - \frac{\Xv^2}{2!} + \frac{\Xv^4}{4!} - \frac{\Xv^6}{6!} + \cdots \right) \\
    &amp; = - \Xv + \frac{\Xv^3}{3!} - \frac{\Xv^5}{5!} + \cdots = - \sin \Xv
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1309" class="slide " data-line="1309" data-h="13" data-v="3">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>迹对矩阵求导</h5></div></div>
<p>

$$
\begin{align*}
    \left[ \frac{\partial \tr(\Uv \Vv)}{\partial \Xv} \right]_{ij} = \frac{\partial \tr(\Uv \Vv)}{\partial x_{ji}} = \tr \left( \frac{\partial (\Uv \Vv)}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>取<span class="mathjax-exps">$\Uv = \Iv$</span>，<span class="mathjax-exps">$\Vv = \Av \otimes \Xv, ~ \Xv \otimes \Xv$</span>，则</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial \tr(\Av \otimes \Xv)}{\partial \Xv} \right]_{ij} = \tr \left( \frac{\partial (\Av \otimes \Xv)}{\partial x_{ji}} \right) = \tr \left( \Av \otimes \frac{\partial \Xv}{\partial x_{ji}} \right) \\
    &amp; ~ = \tr ( \Av \otimes \Ev_{ji} ) = \tr(\Av) \delta_{ij} \Longrightarrow \frac{\partial \tr(\Av \otimes \Xv)}{\partial \Xv} = \tr(\Av) \Iv \\
    \\
    &amp; \left[ \frac{\partial \tr(\Xv \otimes \Xv)}{\partial \Xv} \right]_{ij} = \tr \left( \frac{\partial (\Xv \otimes \Xv)}{\partial x_{ji}} \right) = \tr \left( \frac{\partial \Xv}{\partial x_{ji}} \otimes \Xv + \Xv \otimes \frac{\partial \Xv}{\partial x_{ji}} \right) \\
    &amp; ~ = \tr ( \Ev_{ji} \otimes \Xv ) + \tr ( \Xv \otimes \Ev_{ji} ) = 2 \tr(\Xv) \delta_{ij} \Longrightarrow \frac{\partial \tr(\Xv \otimes \Xv)}{\partial \Xv} = 2 \tr(\Xv) \Iv
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section><section><section data-notes="" lineno="1341" class="slide " data-line="1341" data-h="14" data-v="0">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>行列式对矩阵求导</h5></div></div>
<p>设<span class="mathjax-exps">$\Xv \in \Rbb^{m \times n}, \Av \in \Rbb^{l \times m}, \Bv \in \Rbb^{n \times l}, \Yv = \Av \Xv \Bv \in \Rbb^{l \times l}$</span>，<span class="mathjax-exps">$\Av$</span>、<span class="mathjax-exps">$\Bv$</span>与<span class="mathjax-exps">$\Xv$</span>无关</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial |\Av \Xv \Bv|}{\partial \Xv} \right]_{ij} = \frac{\partial |\Yv|}{\partial x_{ji}} = \sum_{p,q} \frac{\partial |\Yv|}{\partial y_{pq}}\frac{\partial y_{pq}}{\partial x_{ji}} = \tr \left( \frac{\partial |\Yv|}{\partial \Yv} \frac{\partial \Yv}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>记<span class="mathjax-exps">$y_{ji}$</span>有微小增量<span class="mathjax-exps">$\epsilon$</span>后的矩阵为<span class="mathjax-exps">$\Yv(y_{ji} + \epsilon)$</span>，根据第<span class="mathjax-exps">$j$</span>行拉普拉斯展开</p>
<p>

$$
\begin{align*}
    |\Yv(y_{ji} + \epsilon)| - |\Yv| = \epsilon ~ C_{ji}
\end{align*}
$$
</p>

<p>其中<span class="mathjax-exps">$C_{ji}$</span>是关于<span class="mathjax-exps">$y_{ji}$</span>的<span class="blue">代数余子式</span>，于是</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial |\Yv|}{\partial \Yv} \right]_{ij} &amp; = \frac{\partial |\Yv|}{\partial y_{ji}} = \lim_{\epsilon \rightarrow 0} \frac{|\Yv(y_{ji} + \epsilon)| - |\Yv|}{\epsilon} = C_{ji} ~ \Longrightarrow ~ \frac{\partial |\Yv|}{\partial \Yv} = \Yv^*
\end{align*}
$$
</p>

<p>其中<span class="mathjax-exps">$\Yv^*$</span>是<span class="mathjax-exps">$\Yv$</span>的<span class="blue">伴随矩阵</span> (adjugate matrix)</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1385" class="slide " data-line="1385" data-h="14" data-v="1">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>行列式对矩阵求导</h5></div></div>
<p>设<span class="mathjax-exps">$\Xv \in \Rbb^{m \times n}, \Av \in \Rbb^{l \times m}, \Bv \in \Rbb^{n \times l}, \Yv = \Av \Xv \Bv \in \Rbb^{l \times l}$</span>，<span class="mathjax-exps">$\Av$</span>、<span class="mathjax-exps">$\Bv$</span>与<span class="mathjax-exps">$\Xv$</span>无关</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial |\Av \Xv \Bv|}{\partial \Xv} \right]_{ij} = \frac{\partial |\Yv|}{\partial x_{ji}} = \sum_{p,q} \frac{\partial |\Yv|}{\partial y_{pq}}\frac{\partial y_{pq}}{\partial x_{ji}} = \tr \left( \Yv^* \frac{\partial \Yv}{\partial x_{ji}} \right)
\end{align*}
$$
</p>

<p>又第二项</p>
<p>

$$
\begin{align*}
    \frac{\partial \Yv}{\partial x_{ji}} = \frac{\partial \Av \Xv \Bv}{\partial x_{ji}} = \Av \frac{\partial \Xv}{\partial x_{ji}} \Bv = \Av \Ev_{ji} \Bv
\end{align*}
$$
</p>

<p>代入可得</p>
<p>

$$
\begin{align*}
    \left[ \frac{\partial |\Av \Xv \Bv|}{\partial \Xv} \right]_{ij} &amp; = \tr (\Yv^* \Av \Ev_{ji} \Bv) = [\Bv \Yv^* \Av]_{ij} \\
    &amp; \Longrightarrow \class{blue}{\frac{\partial |\Av \Xv \Bv|}{\partial \Xv} = \Bv (\Av \Xv \Bv)^* \Av}
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1428" class="slide " data-line="1428" data-h="14" data-v="2">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>行列式对矩阵求导</h5></div></div>
<p>设<span class="mathjax-exps">$\Xv \in \Rbb^{m \times n}, \Av \in \Rbb^{l \times m}, \Bv \in \Rbb^{n \times l}, \Yv = \Av \Xv \Bv \in \Rbb^{l \times l}$</span>，<span class="mathjax-exps">$\Av$</span>、<span class="mathjax-exps">$\Bv$</span>与<span class="mathjax-exps">$\Xv$</span>无关</p>
<p>

$$
\begin{align*}
    \class{blue}{\frac{\partial |\Av \Xv \Bv|}{\partial \Xv} = \Bv (\Av \Xv \Bv)^* \Av}
\end{align*}
$$
</p>

<p>若<span class="mathjax-exps">$\Xv, \Av, \Bv$</span>均为可逆方阵，则<span class="mathjax-exps">$\Yv = \Av \Xv \Bv$</span>亦为可逆方阵，于是</p>
<p>

$$
\begin{align*}
    \frac{\partial |\Av \Xv \Bv|}{\partial \Xv} = \Bv (\Av \Xv \Bv)^* \Av = \Bv |\Av \Xv \Bv| (\Av \Xv \Bv)^{-1} \Av = |\Av \Xv \Bv| \Xv^{-1}
\end{align*}
$$
</p>

<p>进一步若<span class="mathjax-exps">$\Av = \Bv = \Iv$</span>，则<span class="mathjax-exps">$\frac{\partial |\Xv|}{\partial \Xv} = \Xv^* = |\Xv| \Xv^{-1}$</span>，由此可得</p>
<p>

$$
\begin{align*}
    \frac{\partial |\Xv^n|}{\partial \Xv} = \frac{\partial |\Xv|^n}{\partial \Xv} = n |\Xv|^{n-1} \Xv^* = n |\Xv|^n \Xv^{-1} = n |\Xv^n| \Xv^{-1}
\end{align*}
$$
</p>

<p>若<span class="mathjax-exps">$a$</span>与<span class="mathjax-exps">$\Xv$</span>无关，则<span class="mathjax-exps">$\frac{\partial \ln |a \Xv|}{\partial \Xv} = \frac{\partial \ln a^m |\Xv|}{\partial \Xv} = \frac{\partial \ln |\Xv|}{\partial \Xv} = \frac{1}{|\Xv|} \frac{\partial |\Xv|}{\partial \Xv} = \frac{\Xv^*}{|\Xv|} = \Xv^{-1}$</span></p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1472" class="slide " data-line="1472" data-h="14" data-v="3">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>行列式对矩阵求导</h5></div></div>
<p>设<span class="mathjax-exps">$\Xv \in \Rbb^{m \times n}, \Av \in \Rbb^{m \times m}, \Yv = \Xv^\top \Av \Xv \in \Rbb^{n \times n}$</span>可逆，<span class="mathjax-exps">$\Av$</span>与<span class="mathjax-exps">$\Xv$</span>无关，易知</p>
<p>

$$
\begin{align*}
    &amp; \left[ \frac{\partial |\Xv^\top \Av \Xv|}{\partial \Xv} \right]_{ij} = \tr \left( \frac{\partial |\Xv^\top \Av \Xv|}{\partial \Yv} \frac{\partial \Yv}{\partial x_{ji}} \right) = \tr \left( \Yv^* \frac{\partial \Xv^\top \Av \Xv}{\partial x_{ji}} \right) \\
    &amp; = \tr \left( \Yv^* \frac{\partial \Xv^\top}{\partial x_{ji}} \Av \Xv \right) + \tr \left( \Yv^* \Xv^\top \Av \frac{\partial \Xv}{\partial x_{ji}} \right) \\
    &amp; = \tr ( \Yv^* \Ev_{ij} \Av \Xv ) + \tr ( \Yv^* \Xv^\top \Av \Ev_{ji} ) = [\Av \Xv \Yv^*]_{ji} + [\Yv^* \Xv^\top \Av]_{ij}
\end{align*}
$$
</p>

<p>于是</p>
<p>

$$
\begin{align*}
    \frac{\partial |\Xv^\top \Av \Xv|}{\partial \Xv} &amp; = (\Av \Xv \Yv^*)^\top + \Yv^* \Xv^\top \Av \\
    &amp; = (\Av \Xv |\Xv^\top \Av \Xv| (\Xv^\top \Av \Xv)^{-1})^\top + |\Xv^\top \Av \Xv| (\Xv^\top \Av \Xv)^{-1} \Xv^\top \Av \\
    &amp; = |\Xv^\top \Av \Xv| (\Xv^\top \Av^\top \Xv)^{-1} \Xv^\top \Av^\top + |\Xv^\top \Av \Xv| (\Xv^\top \Av \Xv)^{-1} \Xv^\top \Av \\
    &amp; = |\Xv^\top \Av \Xv| ((\Xv^\top \Av^\top \Xv)^{-1} \Xv^\top \Av^\top + (\Xv^\top \Av \Xv)^{-1} \Xv^\top \Av)
\end{align*}
$$
</p>

<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section><section vertical="true" data-notes="" lineno="1507" class="slide " data-line="1507" data-h="14" data-v="4">
<div class="header"><img class="hust" src=""><div class="title"><hr class="hr_top"><h5>行列式对矩阵求导</h5></div></div>
<p>如果<span class="mathjax-exps">$\Av$</span>对称，则</p>
<p>

$$
\begin{align*}
    \frac{\partial |\Xv^\top \Av \Xv|}{\partial \Xv} = 2 |\Xv^\top \Av \Xv| (\Xv^\top \Av \Xv)^{-1} \Xv^\top \Av
\end{align*}
$$
</p>

<p>若<span class="mathjax-exps">$\Xv$</span>、<span class="mathjax-exps">$\Av$</span>是方阵，则其均可逆，于是</p>
<p>

$$
\begin{align*}
    \frac{\partial |\Xv^\top \Av \Xv|}{\partial \Xv} = 2 |\Xv^\top| |\Av| |\Xv| \Xv^{-1} \Av^{-1} \Xv^{-\top} \Xv^\top \Av = 2 |\Xv|^2 |\Av| \Xv^{-1}
\end{align*}
$$
</p>

<p>若<span class="mathjax-exps">$\Av = \Iv$</span>，则</p>
<p>

$$
\begin{align*}
    \frac{\partial |\Xv^\top \Xv|}{\partial \Xv} = 2 |\Xv^\top \Xv| (\Xv^\top \Xv)^{-1} \Xv^\top = 2 |\Xv^\top \Xv| \Xv^\dagger, \quad \frac{\partial \ln |\Xv^\top \Xv|}{\partial \Xv} = 2 \Xv^\dagger
\end{align*}
$$
</p>

<p>其中<span class="mathjax-exps">$\Xv^\dagger$</span>是<span class="mathjax-exps">$\Xv$</span>的<span class="blue">伪逆</span> (pseudoinverse)</p>
<div class="footer"><hr class="hr_bottom"><div class="multi_column"><h6 class="bottom_left">图神经网络导论</h6><h6 class="bottom_center">矩阵求导</h6><h6 class="bottom_right">tengzhang@hust.edu.cn</h6></div></div>
</section></section>
      </div>
    </div>
    
      </div>
      
      
    
    
      <script>
        Reveal.initialize({"margin":0,"transition":"none","enableSpeakerNotes":true,"dependencies":[{"src":"../common/js/notes/notes.js","async":true}]})
      </script>
      
    
    
    
    
    
    
  
    </body></html>